Storing related accounts data in the TOML files. Visualization. NetworkX graph

Until this point, i stored accounts and their secret data in .ini, .csv, .txt, .pem files and etc., for example credentials, access keys, certificates. Many simple things can be described by and stored in those file types, but accounts have been getting more and more interrelated and their secret data changed frequently. In such a case, having some typed data, accounts references and sub-properties becomes a necessity and requires me to take somewhat capable of providing it. TOML is capable of.

Basically, .toml file is a tree structure document that can contain strings, integers, booleans, lists, and dictionaries. All of them extend the capabilities of storing multiple account data of a service in one file. Two other benefits i find very handy about TOML are Python 3.11 includes the toml parser in the standard library and its ABNF specification is around 250 lines.

Accounts interrelation. That's quite natural when many services are used with multiple accounts. These accounts can arbitrarily relate to each other. With TOML specification a relation could be a key with a string value or list of strings of filenames.

In regard to all those features, i need some automation to check the relations for integrity and reduce manual check operations, at least. So, here below i share what i've implemented and its demonstration.

References and a checker

The relations are references to the related files as a string value or a values list. A few minimalistic rules, as a toml file is a tree structure the references can be assigned for each node. Duplications are allowed.

Here is the .toml sample to depict the idea:

reference = "path/to/email.toml"

[sub-section]
references = ["path/to/creds.csv", "path/to/cert.pem"]

[sub-section-ext]
references = ["path/to/email.toml"]

This is the checker references.py . It scans a folder for the toml files, reads them with toml , follows the references and checks for ref files to exist. Two kinds of commands to run with typer , a text output for a quick check and a visual output, a networkx graph on matplotlib , to understand the accounts structure and see any issues.

Demonstration

Steps from the very basic to the point when errors occur.

Accounts data folder

The tree output of a sample folder with interrelated accounts in the toml files which i scanned by the checker.

$ tree secrets
secrets
├── acc1.toml
├── acc2.toml
├── acc3.toml
...
├── accs.csv
├── backup-codes.txt
├── backup-key.txt
├── fin2.toml
├── lic.toml
├── mail2.toml
├── mail-pri.asc
├── mail-sec.asc
├── mail.toml
├── mobile.toml
└── subs.csv

Text output

Basically, for fast and regular reference checking a text output is enough to see a current integrity state.

$ python references.py check secrets
[info     ] Reference exists.              node=PosixPath('secrets/acc15.toml') reference=mail.toml
[info     ] Reference exists.              node=PosixPath('secrets/acc6.toml') reference=mobile.toml
[info     ] Reference exists.              node=PosixPath('secrets/mail.toml') reference=mail-pri.asc
[info     ] Reference exists.              node=PosixPath('secrets/acc4.toml') reference=backup-codes.txt
...
[info     ] Success.

It the best case, when the state is fine an output shouldn't be read but signals success.

Visual graph output

Along with it, a text output isn't good at understanding relations structure while visualizing it as a graph is. Not much interpretation is needed when you see what the nodes are, where, and how connected.

For a first step, it is worth visualizing the entire picture of the sample folder.

$ python references.py create-plot secrets
$ python references.py create-plot --relations secrets

Both appearances are chaotic even with the relation edges. Color differs on a file type. Purple is the toml type and the others are cyan. It's clear here to conclude that a node on the bottom right is the most related.

The first step depicted how the accounts are interrelated. Next, two improvements are added, the Kamada Kawai nodes layout to group them all around the centers and emphasizing by increasing their size.

$ python references.py create-plot --realtions --layout "kamada kawai" secrets
$ python references.py create-plot --relations --layout "kamada kawai" --emphasize secrets

Ok, it looks meaningful with the center nodes emphasized, especially on the right picture.

Now, as the nodes look anonymous the picture doesn't tell what that most related account is and a few others. Their names could refine detalization. Displaying the names.

$ python references.py create-plot --relations --layout "kamada kawai" --emphasize --names secrets

Nice, it looks much better than text output.

Errors output

The final demo step. If i mistype a reference or forget to update the edited ones the checker points it out to me with the text and visual outputs as below.

$ python references.py create-plot --relations --layout "kamada kawai" --emphasize --names secrets

[error    ] Reference doesn't exist.       node=PosixPath('secrets/acc4.toml') reference=outdated-codes.txt
[error    ] Reference doesn't exist.       node=PosixPath('secrets/mail2.toml') reference=removed-already.toml
[error    ] Reference doesn't exist.       node=PosixPath('secrets/mail.toml') reference=old-mail.toml
[error    ] Reference doesn't exist.       node=PosixPath('secrets/mobile.toml') reference=wrong-acc.toml
[error    ] Fail.

Great, eyes now stick to the red dots and fixing the errors displayed is going to be easier.

Afterward

Having on hand the checker lets me not keep in mind all the accounts relations and preserve their integrity.

reference-checker repository.