First of all, keyfile writer (and reader) are supposed to be able to store
every profile to disk and re-read a valid profile back. Note that the profile
might be modified in the process, for example, blob certificates are written
to a file. So, the result might no be exactly the same, but it must still be
valid (and should only diverge in expected ways from the original, like mangled
certificates).
Previously, we would re-read the profile after writing to disk. If that failed,
we would only fail an assertion but otherwise proceeed. It is a bug
after all. However, it's bad to check only after writing to file,
because it results in a unreadable profile on disk, and in the first
moment it appears that noting went wrong. Instead, we should fail early.
Note that nms_keyfile_reader_from_keyfile() must entirely operate on the in-memory
representation of the keyfile. It must not actually access any files on disk. Hence,
moving this check before writing the profile must work. Otherwise, that would be
a separate bug. Actually, keyfile reader and writer violate this. I
added FIXME comments for that. But it doesn't interfere with this
patch.
(cherry picked from commit 3b8aab2999)
Before, the .nmmeta file could only contain one piece of information:
the loaded-path. This was persisted to disk by writing a "$UUID.nmmeta"
symlink that links to the loaded-path. Also, in practice this is used
for tombstones, so the only valid loaded-path is "/dev/null" (all other
paths are ignored).
Extend the .nmmeta file format to also be able to store additional data: the
shadowed-storage path. We will need that later but the idea is that if
we have a tombstone on disk, then this tombstone might explicitly shadow
another file. The use is when re-adding a profile with the same UUID, then
the existing storage is used (instead of creating a new file). This will
be necessary with Update2(NM_SETTINGS_UPDATE2_FLAG_IN_MEMORY_DETACHED)
flag. This flag first allows to clone a profile from persistent storage
to a profile in /run. Later, when this profile gets deleted, the
original profile will be left on disk. If the same profile then gets
re-created with AddConnection(), then the original filename must be
taken over again. This is to avoid duplication of profiles on disk.
Note that this piece of information is relevent per-UUID, and as such
it's correct to store it in the .nmmeta file. That is related to the
"shadowed-storage" information that we store in the [.nmmeta] section
of keyfiles.
When we make runtime only changes, we may leave the profile in persistent
storage and store a overlay profile in /run. However, in various cases we need
to remember the original path.
Add code to store and retrieve that path from the keyfile.
Note that this data is written inside the keyfile in /run. That is because
this piece of information really depends on that particular keyfile, and not
on the profile/UUID. That is why we write it to the [.nmmeta] section of the
keyfile and not to the .nmmeta file (which is per-UUID).
This patch only adds the backend to write and load the setting from
disk. It's not yet used.
Currently, meta-data has a very narrow use: as tombstones.
Later, we will need to store additional per UUID meta-data. For example,
when a profile becomes unsaved, we may need to remember the original
filename.
Refactor the code for that. This is for the most part just renaming
and slightly different handling of the fields.
Completely rework how settings plugin handle connections and how
NMSettings tracks the list of connections.
Previously, settings plugins would return objects of (a subtype of) type
NMSettingsConnection. The NMSettingsConnection was tightly coupled with
the settings plugin. That has a lot of downsides.
Change that. When changing this basic relation how settings connections
are tracked, everything falls appart. That's why this is a huge change.
Also, since I have to largely rewrite the settings plugins, I also
added support for multiple keyfile directories, handle in-memory
connections only by keyfile plugin and (partly) use copy-on-write NMConnection
instances. I don't want to spend effort rewriting large parts while
preserving the old way, that anyway should change. E.g. while rewriting ifcfg-rh,
I don't want to let it handle in-memory connections because that's not right
long-term.
--
If the settings plugins themself create subtypes of NMSettingsConnection
instances, then a lot of knowledge about tracking connections moves
to the plugins.
Just try to follow the code what happend during nm_settings_add_connection().
Note how the logic is spread out:
- nm_settings_add_connection() calls plugin's add_connection()
- add_connection() creates a NMSettingsConnection subtype
- the plugin has to know that it's called during add-connection and
not emit NM_SETTINGS_PLUGIN_CONNECTION_ADDED signal
- NMSettings calls claim_connection() which hocks up the new
NMSettingsConnection instance and configures the instance
(like calling nm_settings_connection_added()).
This summary does not sound like a lot, but try to follow that code. The logic
is all over the place.
Instead, settings plugins should have a very simple API for adding, modifying,
deleting, loading and reloading connections. All the plugin does is to return a
NMSettingsStorage handle. The storage instance is a handle to identify a profile
in storage (e.g. a particular file). The settings plugin is free to subtype
NMSettingsStorage, but it's not necessary.
There are no more events raised, and the settings plugin implements the small
API in a straightforward manner.
NMSettings now drives all of this. Even NMSettingsConnection has now
very little concern about how it's tracked and delegates only to NMSettings.
This should make settings plugins simpler. Currently settings plugins
are so cumbersome to implement, that we avoid having them. It should not be
like that and it should be easy, beneficial and lightweight to create a new
settings plugin.
Note also how the settings plugins no longer care about duplicate UUIDs.
Duplicated UUIDs are a fact of life and NMSettings must handle them. No
need to overly concern settings plugins with that.
--
NMSettingsConnection is exposed directly on D-Bus (being a subtype of
NMDBusObject) but it was also a GObject type provided by the settings
plugin. Hence, it was not possible to migrate a profile from one plugin to
another.
However that would be useful when one profile does not support a
connection type (like ifcfg-rh not supporting VPN). Currently such
migration is not implemented except for migrating them to/from keyfile's
run directory. The problem is that migrating profiles in general is
complicated but in some cases it is important to do.
For example checkpoint rollback should recreate the profile in the right
settings plugin, not just add it to persistent storage. This is not yet
properly implemented.
--
Previously, both keyfile and ifcfg-rh plugin implemented in-memory (unsaved)
profiles, while ifupdown plugin cannot handle them. That meant duplication of code
and a ifupdown profile could not be modified or made unsaved.
This is now unified and only keyfile plugin handles in-memory profiles (bgo #744711).
Also, NMSettings is aware of such profiles and treats them specially.
In particular, NMSettings drives the migration between persistent and non-persistent
storage.
Note that a settings plugins may create truly generated, in-memory profiles.
The settings plugin is free to generate and persist the profiles in any way it
wishes. But the concept of "unsaved" profiles is now something explicitly handled
by keyfile plugin. Also, these "unsaved" keyfile profiles are persisted to file system
too, to the /run directory. This is great for two reasons: first of all, all
profiles from keyfile storage in fact have a backing file -- even the
unsaved ones. It also means you can create "unsaved" profiles in /run
and load them with `nmcli connection load`, meaning there is a file
based API for creating unsaved profiles.
The other advantage is that these profiles now survive restarting
NetworkManager. It's paramount that restarting the daemon is as
non-disruptive as possible. Persisting unsaved files to /run improves
here significantly.
--
In the past, NMSettingsConnection also implemented NMConnection interface.
That was already changed a while ago and instead users call now
nm_settings_connection_get_connection() to delegate to a
NMSimpleConnection. What however still happened was that the NMConnection
instance gets never swapped but instead the instance was modified with
nm_connection_replace_settings_from_connection(), clear-secrets, etc.
Change that and treat the NMConnection instance immutable. Instead of modifying
it, reference/clone a new instance. This changes that previously when somebody
wanted to keep a reference to an NMConnection, then the profile would be cloned.
Now, it is supposed to be safe to reference the instance directly and everybody
must ensure not to modify the instance. nmtst_connection_assert_unchanging()
should help with that.
The point is that the settings plugins may keep references to the
NMConnection instance, and so does the NMSettingsConnection. We want
to avoid cloning the instances as long as they are the same.
Likewise, the device's applied connection can now also be referenced
instead of cloning it. This is not yet done, and possibly there are
further improvements possible.
--
Also implement multiple keyfile directores /usr/lib, /etc, /run (rh #1674545,
bgo #772414).
It was always the case that multiple files could provide the same UUID
(both in case of keyfile and ifcfg-rh). For keyfile plugin, if a profile in
read-only storage in /usr/lib gets modified, then it gets actually stored in
/etc (or /run, if the profile is unsaved).
--
While at it, make /etc/network/interfaces profiles for ifupdown plugin reloadable.
--
https://bugzilla.gnome.org/show_bug.cgi?id=772414https://bugzilla.gnome.org/show_bug.cgi?id=744711https://bugzilla.redhat.com/show_bug.cgi?id=1674545
The file got a wider scope to contain generic meta data about profiles.
Rename the internal API to reflect that (and be consistend with the
naming of the files).
We may want to store meta-data for a profile to disk. The immediate
need are "tombstones": markers that the particular UUID is shadowed
and the profile does not exist (despite being in read-only location).
Change the filename of these symlinks from
".loaded-${UUID}.nmconnection"
to
"${UUID}.nmmeta"
The leading dot is not desirable as tools tend to hide such files.
Use a different scheme for the filename that does not have the leading dot.
Note that nm_keyfile_utils_ignore_filename() would also ignore ".nmmeta"
as not a valid keyfile. This is just what we want, and influences the
choice of this file suffix.
Also, "nmmeta" is a better name, because this name alludes that there is
a wider use for the file: namely to have addtional per-profile metadata.
That is regardless that the upcoming first use will be only to store symlinks
to "/dev/null" to indicate the tombstones.
Note that per-profile metadata is not new. Currently we write the files
/var/lib/NetworkManager/{seen-bssids,timestamps}
that have a similar purpose. Maybe the content from these files could one
day be migrated to the ".nmmeta" file. The naming scheme would make it
suitable.
The previous logic seems complicated to me. I even think it is wrong.
Rework it, I think this makes sense.
Also, previously the existing path was used if the file didn't exist.
I think that is wrong. If for force a rename, then the filename must
not be used even if the file currently does not exist.
Also add an "allow_filename_cb" argument, to reject filenames that
are blacklisted.
We no longer add these. If you use Emacs, configure it yourself.
Also, due to our "smart-tab" usage the editor anyway does a subpar
job handling our tabs. However, on the upside every user can choose
whatever tab-width he/she prefers. If "smart-tabs" are used properly
(like we do), every tab-width will work.
No manual changes, just ran commands:
F=($(git grep -l -e '-\*-'))
sed '1 { /\/\* *-\*- *[mM]ode.*\*\/$/d }' -i "${F[@]}"
sed '1,4 { /^\(#\|--\|dnl\) *-\*- [mM]ode/d }' -i "${F[@]}"
Check remaining lines with:
git grep -e '-\*-'
The ultimate purpose of this is to cleanup our files and eventually use
SPDX license identifiers. For that, first get rid of the boilerplate lines.
nmtst_get_rand_int() was originally named that way, because it
calls g_rand_int(). But I think if a function returns an uint32, it
should also be named that way.
Rename.
It's deprecated and off by default for a long time.
It is bad to automatically reload connection profiles. For example, ifcfg
files may consist of multiple files, there is no guarantee that we
pick up the connection when it's fully written.
Just don't do this anymore.
Users should use D-Bus API or `nmcli connection reload` or `nmcli
connection load $FILENAME` to reload profiles from disk.
We already have "libnm-core/tests/test-keyfile.c" from which we build
"test-keyfile".
Our test binaries should be named the following:
- "*/tests/test-*"
- the test binary "*/tests/test-*" should be build from a source file
"*/tests/test-*.c". Meaning: the source's and executable's name should
correspond.
- test binaries should be named uniquely. Also, because older meson
versions don't like having the same binary name more than once.
Rename to avoid the duplicate name.
From the files under "shared/nm-utils" we build an internal library
that provides glib-based helper utilities.
Move the files of that basic library to a new subdirectory
"shared/nm-glib-aux" and rename the helper library "libnm-core-base.la"
to "libnm-glib-aux.la".
Reasons:
- the name "utils" is overused in our code-base. Everything's an
"utils". Give this thing a more distinct name.
- there were additional files under "shared/nm-utils", which are not
part of this internal library "libnm-utils-base.la". All the files
that are part of this library should be together in the same
directory, but files that are not, should not be there.
- the new name should better convey what this library is and what is isn't:
it's a set of utilities and helper functions that extend glib with
funcitonality that we commonly need.
There are still some files left under "shared/nm-utils". They have less
a unifying propose to be in their own directory, so I leave them there
for now. But at least they are separate from "shared/nm-glib-aux",
which has a very clear purpose.
(cherry picked from commit 80db06f768)
The defaults for test timeouts in meson is 30 seconds. That is not long
enough when running
$ NMTST_USE_VALGRIND=1 ninja -C build test
Note that meson supports --timeout-multiplier, and automatically
increases the timeout when running under valgrind. However, meson
does not understand that we are running tests under valgrind via
NMTST_USE_VALGRIND=1 environment variable.
Timeouts are really not expected to be reached and are a mean of last
resort. Hence, increasing the timeout to a large value is likely to
have no effect or to fix test failures where the timeout was too rigid.
It's unlikely that the test indeed hangs and the increase of timeout
causes a unnecessary increase of waittime before aborting.
- use gs_free instead of explicit free().
- use nm_streq*() instead of strcmp().
- move deletion of existing file after we successfully wrote
the new file.
- add parameter existing_path_readonly, to avoid to overwrite or
delete the existing path (if it exists). This is still mostly unused,
but will be necessary when we have read-only directories.
The 'number' property in GSM settings is a legacy thing that comes
from when ModemManager used user-provided numbers, if any, to connect
3GPP modems.
Since ModemManager 1.0, this property is completely unused for 3GPP
modems, and so it doesn't make sense to use it in the NetworkManager
settings. Ofono does not use it either.
For AT+PPP-based 3GPP modems, the 'number' to call to establish the
data connection is decided by ModemManager itself, e.g. for standard
GSM/UMTS/LTE modems it will connect a given predefined PDP context,
and for other modems like Iridium it will have the number to call
hardcoded in the plugin itself.
https://github.com/NetworkManager/NetworkManager/pull/261
This code will be used later.
We want to remember which keyfiles are currently loaded (or hidden).
With the addition or multiple keyfile directories (soon), there are
two cases where this matters:
- if there are multiple keyfiles which reference the same UUID,
we can only load one of them. That is already a problem today
with only one keyfile directory, where multiple files can reference
the same UUID.
The implementation will pick the file based on priorities (like
the file modification date). However, the user may call explicitly
call `nmcli connection load`. In that case, we cannot reload
all files to find out whether the to be loaded file is hidden
according to the defined priorities. We cannot do that, because we
must not make decisions based on files on disk, which we are not told
to reload. So, during a `nmcli connection load` we must look at
unrelated files, to determine how to load the file.
Instead, we do allow the user to load any file, even if it would be
shadowed by other files. When we do that, we may want to persist which
file is currently loaded, so that a service restart and a `nmcli connection
reload` does not undo the load again. This can be later later be solved by
writing a symlink
"/var/run/NetworkManager/system-connections/.loaded-$UUID.nmkeyfile"
which targets the currently active file.
- if a profile was loaded from read-only persistant storage, the user
may still delete the profile. We also need to remember the deletion
of the file. That will be achieved by symlinking "/dev/null" as
"/etc/NetworkManager/system-connections/.loaded-$UUID.nmkeyfile".
Add helper functions to read and write these symlinks.
Correct the spelling across the *entire* tree, including translations,
comments, etc. It's easier that way.
Even the places where it's not exposed to the user, such as tests, so
that we learn how is it spelled correctly.
For upstream, we changed behavior here. However, I think certain
downstream don't want to do that, and revert patch "d37ad15f12 keyfile:
also add ".nmconnection" extension when writing keyfiles in /etc".
For that to make easier, keep the upstream sources closer to what
was. Revert.
This reverts commit e93d8cdb74.
NM_CONFIG_KEYFILE_PATH_IN_MEMORY is now called NMS_KEYFILE_PATH_NAME_RUN.
This name seems odd in the current context, it will be more suitable
when we also have NMS_KEYFILE_PATH_NAME_LIB (for /usr/lib).
These utilities are concerned with valid file names (as NetworkManager
daemon requires it). This is relevant for everybody who wants to write
keyfile files directly. Hence, move it to libnm-core. Still as internal
API.
For profiles in "/etc/NetworkManager/system-connections", we did not enforce
that the keyfiles have a special suffix, nor did we generate the
filenames in such a manner. In hindsight, I think that was a mistake.
Recently we added "/run/NetworkManager/system-connections" as additional
keyfile directory. Enforce a suffix and write keyfiles with such a name.
In principle, we could also start writing keyfiles in /etc with the
same suffix. But let's not do that, because we anyway cannot enforce
it.
An ugly part is, that during `nmcli connection load` we need to
determine whether the to-be-loaded connection is under /etc or /run.
Preferably, we would allow any kind of symlinking as what matters
is the file object (inode) and not the path. Anyway, we don't do
that but compare plain paths. That means, paths which are not
in an expected form, will be rejected. In particular, the paths
starting with "/run/..." and "/var/run/..." will be treated differently,
and one of them will be rejected.
Note that ifcfg-rh plugin strictly enforces that the path
starts with IFCFG_DIR as well. So, while this is a breaking
change for keyfile, I think it's reasonable.
keyfile already supports omitting the "connection.id" and
"connection.uuid". In that case, the ID would be taken from the
keyfile's name, and the UUID was generated by md5 hashing the
full filename.
No longer do this during nm_keyfile_read(), instead let all
callers call nm_keyfile_read_ensure_*() to their liking. This is done
for two reasons:
- a minor reason is, that one day we want to expose keyfile API
as public API. That means, we also want to read keyfiles from
stdin, where there is no filename available. The implementation
which parses stdio needs to define their own way of auto-generating
ID and UUID. Note how nm_keyfile_read()'s API no longer takes a
filename as argument, which would be awkward for the stdin case.
- Currently, we only support one keyfile directory, which (configurably)
is "/etc/NetworkManager/system-connections".
In the future, we want to support multiple keyfile dirctories, like
"/var/run/NetworkManager/profiles" or "/usr/lib/NetworkManager/profiles".
Here we want that a file "foo" (which does not specify a UUID) gets the
same UUID regardless of the directory it is in. That seems better, because
then the UUID won't change as you move the file between directories.
Yes, that means, that the same UUID will be provided by multiple
files, but NetworkManager must already cope with that situation anyway.
Unfortunately, the UUID generation scheme hashes the full path. That
means, we must hash the path name of the file "foo" inside the
original "system-connections" directory.
Refactor the code so that it accounds for a difference between the
filename of the keyfile, and the profile_dir used for generating
the UUID.
This is useful for in-memory connections to persist NetworkManager
restarts (as opposed to machine restarts).
Perhaps most improtantly, this allows generating in-memory connections outside
NetworkManager, e.g. passing configuration from early boot firmware in initrd.
Note that this does *not* aspire to do more than it says on the tin:
Notably, it doesn't touch the problem of provisioning connections in multiple
persistent connection directories and thus doesn't have to deal with the
problem of deleting or overlaying the connections tha (rh #772414) deals
with.
NMSettingsPlugin was a glib interface, not a regular GObject
instance. Accordingly, settings plugins would implement this interface
instead of subclassing a parent type.
Refactor the code, and make NMSettingsPlugin a GObject type. Plugins
are now required to subclass this type.
Glib interfaces are more cumbersome than helpful. At least, unless
there is a good reason for using them.
Our settings plugins are all internal API and are entirely under
our control. It also means, this change is fine, as there are no
implementations outside of this source tree.
Using interfaces do would allow more flexibility in implementing the
settings plugin.
For example, the plugin would be able to derive from any other GObject
type, like NMKimchiRefrigerator. But why would we even? Let's not add monster
classes that implement house appliances beside NMSettingsPluginInterface.
The settings plugin should have one purpose only: being a settings plugin.
Hence, requiring it to subclass NMSettingsPlugin is more than resonable. We
don't need interfaces for this.
Now that NMSettingsPlugin is a regular object instance, it may also have
state, and potentially could provide common functionality for the plugin
implementation -- if that turns out to be useful. Arguably, an interface can
have state too, for example by attaching the state somewhere else (like
NMConnection does). But let's just say no.
On a minor note, this also avoids some tiny overhead that comes with
glib interfaces.
Previously, nms_keyfile_writer_connection() would only return @out_path, if
it differed from @existing_path. That might make sense, if we could thereby
avoid duplicating @existing_path, however, we never did that
optimization.
Just consistently always return the path, let the caller deal with this.
