Imagine we wait for a device, the device appears and starts activating.
That might take a while (during which it has a pending action). In the
meantime, the "connection.wait-device-timeout" timeout expires.
Now we want to log a warning about profiles that don't have their
device upon timeout. However, that the device is still busy at that
point is irrelevant. Skip logging a message about those profiles.
Fixes: 3df662f534 ('settings: rework wait-device-timeout handling and consider device compatibility')
A profile can configure "connection.wait-device-timeout" to indicate
that startup complete is blocked until a suitable device around.
This is useful for NetworkManager-wait-online and initrd mode.
Previously, we looked at NMPlatform whether a link with matching
interface-name was present. That is wrong because it cannot handle
profiles that rely on "ethernet.mac-address" setting or other "match"
settings. Also, the mere presence of the link does not yet mean
that the NMDevice was created and ready. In fact, there is a race here:
NMPlatform indicates that the device is ready (unblocking NMSettings),
but there is no corresponding NMDevice yet which keeps NetworkManager
busy to block startup complete.
Rework this. Now, only check whether there is a compatible device for
the profile.
Since we wait for compatible devices, it works now not only for the
interface name. Note that we do some optimizations so that we don't have
to re-evaluate all profiles (w.r.t. all devices) whenever something on the
device changes: we only care about this when all devices finally become
ready.
Also, we no longer start the timeout for "connection.wait-device-timeout"
when the profile appears. Instead, there is one system-wide start time
(NMSettingsPrivate.startup_complete_start_timestamp_msec). That simplifies
code and makes sense: we start waiting when NetworkManager is starting, not
when the profile gets added. Also, we wait for all profiles to become
ready together.
NMSettings needs access to the list of all devices, which is tracked
by NMManager. Of course, this ties NMSettings and NMManager closer
together. Note that NMManager already owns a reference to NMSettings,
so they are in fact related.
The alternatives of just letting NMSettings reference NMManager (and
vice versa) would be more complicated, and likely not help to simplify
the code (on the contrary).
When a device is not marked as unmanaged, but also not actively managed
by NetworkManager, then NetworkManager will generate an in-memory
profile to represent the active state, if the device is up and
configured (with an IP address).
Such profiles are commonly named like "eth0", and they are utterly
confusing to users, because they look as if NetworkManager actually
manages the device, when it really just shows that somebody else configures
the device.
We should express this better in the UI, hence add flags to indicate
that.
In practice, such profiles are UNSAVED, NM_GENERATED, and VOLATILE. But
add an explicit flag to represent that.
https://bugzilla.redhat.com/show_bug.cgi?id=1816202
Scenario:
- have ethernet connection as unmanaged.
- create (or have) a suitable profile for the connection.
- make the device as managed. No default wired connection gets
created.
- delete the profile.
Note that NMManager does in manager_device_state_changed():
»···if (NM_IN_SET (new_state,
»··· NM_DEVICE_STATE_UNAVAILABLE,
»··· NM_DEVICE_STATE_DISCONNECTED))
»···»···nm_settings_device_added (priv->settings, device);
that means, when the device the next time goes through
UNAVAILABLE/DISCONNECTED states, we will suddenly create the
default "Wired connection 1" profile.
That doesn't seem right. When a device is suitable to have a
default-wired connection, we should only check once whether to
create it. We should not retry that later. The !no-auto-default
mechanism exists so we can start NetworkManager without a profile for
the device. It doesn't mean that we later one (after previously deciding
not to create a profile), we still create it.
https://bugzilla.redhat.com/show_bug.cgi?id=1687937https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/450
Several macros are used to define function. They had a "_STATIC" variant,
to define the function as static.
I think those macros should not try to abstract entirely what they do.
They should not accept the function scope as argument (or have two
variants per scope). This also because it might make sense to add
additional __attribute__(()) to the function. That only works, if
the macro does not pretend to *not* define a plain function.
Instead, embrace what the function does and let the users place the
function scope as they see fit.
This also follows what is already done with
static NM_CACHED_QUARK_FCN ("autoconnect-root", autoconnect_root_quark)
Keyfile support was initially added under GPL-2.0+ license as part of
core. It was moved to "libnm-core" in commit 59eb5312a5 ('keyfile: merge
branch 'th/libnm-keyfile-bgo744699'').
"libnm-core" is statically linked with by core and "libnm". In
the former case under terms of GPL-2.0+ (good) and in the latter case
under terms of LGPL-2.1+ (bad).
In fact, to this day, "libnm" doesn't actually use the code. The linker
will probably remove all the GPL-2.0+ symbols when compiled with
gc-sections or LTO. Still, linking them together in the first place
makes "libnm" only available under GPL code (despite the code
not actually being used).
Instead, move the GPL code to a separate static library
"shared/nm-keyfile/libnm-keyfile.la" and only link it to the part
that actually uses the code (and which is GPL licensed too).
This fixes the license violation.
Eventually, it would be very useful to be able to expose keyfile
handling via "libnm". However that is not straight forward due to the
licensing conflict.
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/merge_requests/381
Move it to shared as it's useful for clients as well.
Move and rename nm_dbus_manager_new_auth_subject_from_context() and
nm_dbus_manager_new_auth_subject_from_message() in nm-dbus-manager.c
as they're needed there.
The abbreviations "ns" and "ms" seem not very clear to me. Spell them
out to nsec/msec. Also, in parts we already used the longer abbreviations,
so it wasn't consistent.
Note that the server always returns TRUE for the boolean return value
of ReloadConnections. Hence, this should not change in behavior, because
the server would never have returned FALSE.
However, change behavior of the API. It's odd that the function might
return %FALSE without setting the error output. It's also not clear
what the boolean value of the "ReloadConnections" D-Bus would mean
anyway.
have_connection_for_device() really should just call nm_device_check_connection_compatible().
Note that nm_device_check_connection_compatible() of course checks the
connection type already, so this is redundant.
This check is only useful for devices that implement new_default_connection.
We can shortcut the possibly expensive checks like have_connection_for_device(),
which need to iterate all profiles.
If a profile has only "ethernet.mac-address" set, but
"connection.interface-name" not, then the previous check
iface = nm_setting_connection_get_interface_name (s_con);
if (!nm_streq0 (iface, nm_device_get_iface (device)))
continue;
would wrongly consider the profile not matching for the device.
As a result, we would wrongly create a auto-default connection.
Fix that. We already call nm_device_check_connection_compatible()
above. That is fully suitable to compare the interface name and
the MAC address. We don't need to duplicate this check (wrongly).
See also commit 77d01c9094 ('settings: ignore incompatible connections
when looking for existing ones') for how this code changed.
https://bugzilla.redhat.com/show_bug.cgi?id=1727909
Now nm_shutdown_wait_obj_*() supports two styles:
- NM_SHUTDOWN_WAIT_TYPE_OBJECT: this just registers a weak pointer
on a source GObject. As long as the object is not destroyed
(and the object is not unregistered), the shutdown gets blocked.
- now new is NM_SHUTDOWN_WAIT_TYPE_CANCELLABLE: this source object
is a GCancellable, and during shutdown, the system will cancel
the instances to notify about the shutdown. That aside, the GCancellable
is tracked exactly like a regular NM_SHUTDOWN_WAIT_TYPE_OBJECT (meaning:
a weak pointer is registered and shutdown gets delayed as long as the instance
lives).
As the rest of the shutdown, it's not yet implemented on the shutdown-side.
What is now possible is to register such cancellables, so that users can make
use of this API before we fix shutdown. We cannot fix it all at the same time,
so first users must be ready for this approach.
All D-Bus method call implementations use similar error messages when
authenticating requests; add defines for them to ensure the same exact
message is reused.
NM didn't support wpa-none for years because kernel drivers used to be
broken. Note that it wasn't even possible to *add* a connection with
wpa-none because it was rejected in nm_settings_add_connection_dbus().
Given that wpa-none is also deprecated in wpa_supplicant and is
considered insecure, drop altogether any reference to it.
Tombstones in /etc are not only to hide storages of type keyfile. They
are for hiding/shadowing any profile from persistant storage. That's
why we need to compare them already in _sett_conn_entry_sds_update().
Fix the prioriy of storages for the same UUID.
Note that the "$UUID.nmmeta" files (the tombstones) are handled by
keyfile plugin. But that is only to load them together during `nmcli
connection reload` when we iterate the files of the system-connections
directory. For the most part, nmmeta/tombstones are not keyfile specific
and handled by NMSettings. A tombstone in /run hides any profile (regardless
of the settings plugin). And a tombstones in /etc hides any profile, except
in-memory connections from keyfile /run directory.
Via Update2() D-Bus API there are three ways how a profile can be stored
(or migrated) to in-memory:
- NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY
- NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED
- NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_ONLY
With the recent rework of settings I dropped NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY
and it had the same meaning as NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED.
However, the way NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED was implemented is
problematic. The problem is that it leaves the profile on disk but creates an
in-memory representation which shadows the persistent storage. Later,
when storing the profile to disk again, a new filename is chosen.
This allows via D-Bus API to toggle between NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED
and NM_SETTINGS_UPDATE2_FLAG_TO_DISK, and thereby pilling up profiles on disk.
Also, there is no D-Bus API to do anything sensible with these leaked, shadowed
profiles on disk.
Note that if we have a read-only profile in /usr/lib or in ifupdown
plugin, then the problem is not made any worse. That is, because via D-Bus
API such profiles can be made in-memory, and afterwards stored to /etc.
Thereby too the profile gets duplicate on disk, but this game only
works once. Afterwards, you cannot repeat it to create additional
profiles on disk. It means, you can only leak profiles once, and only
if they already exist in read-only storage to begin with.
This problem with NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED already existed
before the settings-delegate-storage rework, and is unrelated to whether in-memory
profiles now happen to be persisted to /run.
Note that NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_ONLY is simple and does not suffer
from this problem. When you move a profile to in-memory-only, it gets deleted from
persistent storage and no duplication happens.
The problem is that NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED used to
forget about the profile that it shadows, and that is wrong.
So, first re-add proper support for NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY. This
works by remembering the "shadowed-storage" path for in-memory profiles.
When later saving such a profile to disk again, the shadowed-storage
will be re-used. Likewise, when deleting such a profile, the shadowed
storage will be deleted.
Note that we keep NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED and it
also remembers the shadowed storage (but without "owning" it). That means,
when such a profile gets saved to disk again, the orginal storage is
reused too. As such, during future updates it behaves just like
NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY. The difference is when deleting
such a profile. In this case, the profile is left on storage and a
tombstone gets written. So, how is this better than before and why even
keep this complicated flag?
First, we keep this flag because we really want the ansible role to be
able to do in-memory changes only. That implies being able to delete a
profile from NetworkManager's view, but not from persistent storage. Without
this flag there is no way to do that. You can only modify an on-disk profile
by shadowing it, but you could not delete it form NetworkManager's view
while keeping it on disk.
The new form of NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED is safe and avoids
the duplication problem because also for tombstones it remembers the original
"shadowed-storage". That is, when the profile gets recreated later via
D-Bus API AddConnection, then the re-created profile will still reference
and reuse the shadowed storage that it had before deletion.
Say, you configure "plugins=ifcfg-rh,keyfile" and have an ifcfg file for
a certain profile. Then you make it IN-MEMORY-DETACHED and delete it.
The result is that the ifcfg file still exists, but there is a tombstone
nmmeta file that shadows the profile.
Now, if you want to re-add the same profile (same UUID) and store it to
persistent storage, then first it will try to persist the profile via
the ifcfg-rh plugin. That may not be possible, for example because the
connection type is not supported by the plugin.
Now, you could write it to /etc as keyfile, but then there would still
be a higher priority profile. Note that after calling
_add_connection_to_first_plugin() we issue
_connection_changed_track (self, new_storage, new_connection, TRUE);
(note the prioritize=TRUE parameter). So, in the first moment, all looks
good. However it is not because upon first reload the change gets
reverted and the other profile resurfaces.
The problem are that all settings plugins (except keyfile) may be
completely unable to persist a profile. The real and only solution is
not to use any settings plugins except keyfile.
In a previous version (that never was merged), the "loaded-path" of nmmeta
files was used to prioritize profiles. Since that was not done, we
should not add profiles that we know have lower priority than existing
profiles.
Before, the .nmmeta file could only contain one piece of information:
the loaded-path. This was persisted to disk by writing a "$UUID.nmmeta"
symlink that links to the loaded-path. Also, in practice this is used
for tombstones, so the only valid loaded-path is "/dev/null" (all other
paths are ignored).
Extend the .nmmeta file format to also be able to store additional data: the
shadowed-storage path. We will need that later but the idea is that if
we have a tombstone on disk, then this tombstone might explicitly shadow
another file. The use is when re-adding a profile with the same UUID, then
the existing storage is used (instead of creating a new file). This will
be necessary with Update2(NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED)
flag. This flag first allows to clone a profile from persistent storage
to a profile in /run. Later, when this profile gets deleted, the
original profile will be left on disk. If the same profile then gets
re-created with AddConnection(), then the original filename must be
taken over again. This is to avoid duplication of profiles on disk.
Note that this piece of information is relevent per-UUID, and as such
it's correct to store it in the .nmmeta file. That is related to the
"shadowed-storage" information that we store in the [.nmmeta] section
of keyfiles.
When we make runtime only changes, we may leave the profile in persistent
storage and store a overlay profile in /run. However, in various cases we need
to remember the original path.
Add code to store and retrieve that path from the keyfile.
Note that this data is written inside the keyfile in /run. That is because
this piece of information really depends on that particular keyfile, and not
on the profile/UUID. That is why we write it to the [.nmmeta] section of the
keyfile and not to the .nmmeta file (which is per-UUID).
This patch only adds the backend to write and load the setting from
disk. It's not yet used.
Currently, meta-data has a very narrow use: as tombstones.
Later, we will need to store additional per UUID meta-data. For example,
when a profile becomes unsaved, we may need to remember the original
filename.
Refactor the code for that. This is for the most part just renaming
and slightly different handling of the fields.
It should be possible to add a profile with autoconnect blocked form the
start. Update2() has a %NM_SETTINGS_UPDATE2_FLAG_BLOCK_AUTOCONNECT flag to
block autoconnect, and so we need something similar when adding a connection.
As the existing AddConnection() and AddConnectionUnsaved() API is not
extensible, add AddConnection2() that has flags and room for additional
arguments.
Then add and implement the new flag %NM_SETTINGS_ADD_CONNECTION2_FLAG_BLOCK_AUTOCONNECT
for AddConnection2().
Note that libnm's nm_client_add_connection2() API can completely replace
the existing nm_client_add_connection_async() call. In particular, it
will automatically prefer to call the D-Bus methods AddConnection() and
AddConnectionUnsaved(), in order to work with server versions older than
1.20. The purpose of this is that when upgrading the package, the
running NetworkManager might still be older than the installed libnm.
Anyway, so since nm_client_add_connection2_finish() also has a result
output, the caller needs to decide whether he cares about that result.
Hence it has an argument ignore_out_result, which allows to fallback to
the old API. One might argue that a caller who doesn't care about the
output results while still wanting to be backward compatible, should
itself choose to call nm_client_add_connection_async() or
nm_client_add_connection2(). But instead, it's more convenient if the
new function can fully replace the old one, so that the caller does not
need to switch which start/finish method to call.
https://bugzilla.redhat.com/show_bug.cgi?id=1677068
nm_config_set_no_auto_default_for_device() is called by NMSettings,
so it makes sense that also NMSettings checks whether the device is
blocked.
Of course, there is little difference in practice.
The only downside is that most device types don't implement
new_default_connection(). So the previous form performed the
cheaper check first. On the other hand, we do expect to have
profiles for the devices anyway.
Completely rework how settings plugin handle connections and how
NMSettings tracks the list of connections.
Previously, settings plugins would return objects of (a subtype of) type
NMSettingsConnection. The NMSettingsConnection was tightly coupled with
the settings plugin. That has a lot of downsides.
Change that. When changing this basic relation how settings connections
are tracked, everything falls appart. That's why this is a huge change.
Also, since I have to largely rewrite the settings plugins, I also
added support for multiple keyfile directories, handle in-memory
connections only by keyfile plugin and (partly) use copy-on-write NMConnection
instances. I don't want to spend effort rewriting large parts while
preserving the old way, that anyway should change. E.g. while rewriting ifcfg-rh,
I don't want to let it handle in-memory connections because that's not right
long-term.
--
If the settings plugins themself create subtypes of NMSettingsConnection
instances, then a lot of knowledge about tracking connections moves
to the plugins.
Just try to follow the code what happend during nm_settings_add_connection().
Note how the logic is spread out:
- nm_settings_add_connection() calls plugin's add_connection()
- add_connection() creates a NMSettingsConnection subtype
- the plugin has to know that it's called during add-connection and
not emit NM_SETTINGS_PLUGIN_CONNECTION_ADDED signal
- NMSettings calls claim_connection() which hocks up the new
NMSettingsConnection instance and configures the instance
(like calling nm_settings_connection_added()).
This summary does not sound like a lot, but try to follow that code. The logic
is all over the place.
Instead, settings plugins should have a very simple API for adding, modifying,
deleting, loading and reloading connections. All the plugin does is to return a
NMSettingsStorage handle. The storage instance is a handle to identify a profile
in storage (e.g. a particular file). The settings plugin is free to subtype
NMSettingsStorage, but it's not necessary.
There are no more events raised, and the settings plugin implements the small
API in a straightforward manner.
NMSettings now drives all of this. Even NMSettingsConnection has now
very little concern about how it's tracked and delegates only to NMSettings.
This should make settings plugins simpler. Currently settings plugins
are so cumbersome to implement, that we avoid having them. It should not be
like that and it should be easy, beneficial and lightweight to create a new
settings plugin.
Note also how the settings plugins no longer care about duplicate UUIDs.
Duplicated UUIDs are a fact of life and NMSettings must handle them. No
need to overly concern settings plugins with that.
--
NMSettingsConnection is exposed directly on D-Bus (being a subtype of
NMDBusObject) but it was also a GObject type provided by the settings
plugin. Hence, it was not possible to migrate a profile from one plugin to
another.
However that would be useful when one profile does not support a
connection type (like ifcfg-rh not supporting VPN). Currently such
migration is not implemented except for migrating them to/from keyfile's
run directory. The problem is that migrating profiles in general is
complicated but in some cases it is important to do.
For example checkpoint rollback should recreate the profile in the right
settings plugin, not just add it to persistent storage. This is not yet
properly implemented.
--
Previously, both keyfile and ifcfg-rh plugin implemented in-memory (unsaved)
profiles, while ifupdown plugin cannot handle them. That meant duplication of code
and a ifupdown profile could not be modified or made unsaved.
This is now unified and only keyfile plugin handles in-memory profiles (bgo #744711).
Also, NMSettings is aware of such profiles and treats them specially.
In particular, NMSettings drives the migration between persistent and non-persistent
storage.
Note that a settings plugins may create truly generated, in-memory profiles.
The settings plugin is free to generate and persist the profiles in any way it
wishes. But the concept of "unsaved" profiles is now something explicitly handled
by keyfile plugin. Also, these "unsaved" keyfile profiles are persisted to file system
too, to the /run directory. This is great for two reasons: first of all, all
profiles from keyfile storage in fact have a backing file -- even the
unsaved ones. It also means you can create "unsaved" profiles in /run
and load them with `nmcli connection load`, meaning there is a file
based API for creating unsaved profiles.
The other advantage is that these profiles now survive restarting
NetworkManager. It's paramount that restarting the daemon is as
non-disruptive as possible. Persisting unsaved files to /run improves
here significantly.
--
In the past, NMSettingsConnection also implemented NMConnection interface.
That was already changed a while ago and instead users call now
nm_settings_connection_get_connection() to delegate to a
NMSimpleConnection. What however still happened was that the NMConnection
instance gets never swapped but instead the instance was modified with
nm_connection_replace_settings_from_connection(), clear-secrets, etc.
Change that and treat the NMConnection instance immutable. Instead of modifying
it, reference/clone a new instance. This changes that previously when somebody
wanted to keep a reference to an NMConnection, then the profile would be cloned.
Now, it is supposed to be safe to reference the instance directly and everybody
must ensure not to modify the instance. nmtst_connection_assert_unchanging()
should help with that.
The point is that the settings plugins may keep references to the
NMConnection instance, and so does the NMSettingsConnection. We want
to avoid cloning the instances as long as they are the same.
Likewise, the device's applied connection can now also be referenced
instead of cloning it. This is not yet done, and possibly there are
further improvements possible.
--
Also implement multiple keyfile directores /usr/lib, /etc, /run (rh #1674545,
bgo #772414).
It was always the case that multiple files could provide the same UUID
(both in case of keyfile and ifcfg-rh). For keyfile plugin, if a profile in
read-only storage in /usr/lib gets modified, then it gets actually stored in
/etc (or /run, if the profile is unsaved).
--
While at it, make /etc/network/interfaces profiles for ifupdown plugin reloadable.
--
https://bugzilla.gnome.org/show_bug.cgi?id=772414https://bugzilla.gnome.org/show_bug.cgi?id=744711https://bugzilla.redhat.com/show_bug.cgi?id=1674545
nm_connection_verify() returns success for fully valid (normalized)
connections and also connections that are NM_SETTING_VERIFY_NORMALIZABLE.
We really want to fully normalize the profiles during add-and-activate.
The settings plugins are created by NMSettings when the plugin
gets loaded. There is no need for these instances to be singletons
or to have a singleton getter.
Also, while in practice we create a settings plugin instance of
each type only once, there is nothing that would prevent creating
multiple instances. Hence, having a singleton getter is not right.
What is however useful, is to track them and block shutdown
via nm_shutdown_wait_obj_register*(). While the actual waiting
is not yet implemented, we should mark the plugin instances to
block shutdown (in the future).
The functionality of the ibft settings plugin is now handled by
nm-initrd-generator. There is no need for it anymore, drop it.
Note that ibft called iscsiadm, which requires CAP_SYS_ADMIN to work
([1]). We really want to drop this capability, so the current solution
of a settings plugin (as it is implemented) is wrong. The solution
instead is nm-initrd-generator.
Also, on Fedora the ibft was disabled and probably on most other
distributions as well. This was only used on RHEL.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1371201#c7
