Code

13 pages about "Code"

16 Ideas of Graph Visualization / Graph Databases Usages

I’m passionate about devtool and data visualization for a long time. I’m regularly giving trials to new tools in the mission to improve my productivity; I’m also creating some tools by myself (assh, depviz, wookie ADN solver, etc.)

There is a sub-topic where I’m more consuming external tools than producing ones: graph visualization; which contains itself some sub-topics: graph databases, graph optimizations, social graphs, real-time graph, graph UI, etc.

For about a week now, thinking (again) about ”how to represent a complex system”, and more precisely: ”how to make a collaborative tool that allows to defining, viewing and analyzing an unstructured, complex, evolutive, and living system.”

For now, I’m mostly reading articles and PoCing, from what I can see, there are a lot (maybe too much) of different existing solutions that handles every part I would need for the final solution.

Problems to address

  • how to store the data
  • how to programmatically inject and edit data
  • how to manually inject and edit data
  • how to generate code based on the model definitions
  • how to visualize easily / navigate
  • how to perform queries
  • how to create real-time dashboards

My plan is now to give a more in-depth look at my favorite options. In this intention, I listed 10 (actually 11) ideas of usages that are easier to implement than my target.

Usages for myself

  1. Visualize GitHub issues relationships – dependencies, author, people working on, people commenting out, project & organization hierarchy, milestone grouping, labels tagging, etc.
  2. Visualize Git code/PRs/Commits – similar to GitHub issues above, but based on code instead of issues: code, files, languages, author, PRs, commits, etc.
  3. Define and visualize IT architecture (intermediary milestone to my target) – host, container, ports, cluster, dependencies, Datacenter, process, developers, product, etc.
  4. Personal CRM – maintain a wiki about my relationships (people and company), visualize my social graph, etc.
  5. Blog content relationship – analyze content based on multiple criteria (keywords, tags, labels, label’s metadata, etc.) to analyze what I talk most, and generate better “related posts” suggestions.
  6. Cross-service social graph – create aggregators to visualize people, groups, the friends of friends, followers, followed, etc.
  7. Real-time monitoring – define services hierarchically and then write probs that to monitor the health
  8. Service/Application comparison based on features and other attributes
  9. My GitHub stats; repos, organizations, languages, libraries, metadata, CI used, followers, custom flags (more than 1000 commits, edited < 1 year ago, has a Dockerfile, contributors, etc.).
  10. Log parser to analyze user agent to endpoint (real-time dependency) – https://link.medium.com/rSLv1KGPnU
  11. P2P network efficiency analysis

Bonus: usages for friends

  1. PayFit: engine rules visualization
  2. Doctrine: analysis of legal case relationships
  3. Zenly: social graph, party recommendation
  4. Sounds.am: social graph, friend/playlist/artist/song recommendations
  5. Scaleway: visualize relationships between image / volume / volume layer / server entities

PS: about 1 year ago, I forced myself to list “10 something” every morning for a month. Writing this blog post motivated me to retry the experience and share some outputs in the form of small articles on this blog.

Note: this article is the output of a daily routine, the content of this list won't change over time. It's, however, possible that I create a whole new list on the same subject as a dedicated new post.

Dockerself - runtime self-dockerizer

Presentation

dockerself is a program that creates a new Docker container, injects itself, and finally runs itself from within Docker.

For the record, I was working on pathwar (a security learning platform) and was trying to design the less-constraining way of creating new levels.

My current best idea is to inject a custom entrypoint when the platform starts a new container, instead of requiring the level developers to adapt their Dockerfile.

Additionally, to the simplicity that is added for a level developer, this pattern also has the advantage of always putting the latest version of the entrypoint, even if the image was not updated for a very long time.

Additional resources

$> man fred - fred man page

Image made with Carbon

A good friend is starting his “piscine” at 42 today; it’s only his first day but he already made me the good old joke about “there is no man fred”.

If this joke doesn’t make you smile, this is probably because you don’t know what is the man command for developers and how much useful it is.

Explanation: man is the short for manual, when you type man something in a terminal, you can get some information about the command/function/thing.

More info on Wikipedia: man page, RTFM (read the fucking manual)

Unfortunately, as the fred command does not exist by default, running man fred will just raise an error:

$ man fred
No manual entry for fred

As a workaround, I created a manpage for fred four years ago. Today, I just added some install instructions and an example in the README file.

As soon as you install this manpage on your computer, you will be able to type man fred and have usefulvital! information about me :)

Good luck William for your piscine!

sshportal - An Opinionated SSH Jump-Host

There are many reasons why spaceships don’t have doors that open into the outer space. For one, when a crewmate returns to the ship after visiting a planet, we can do some bio-scanning on them in the airlock, to see if it’s safe to open the doors and let them aboard, or if there’s an alien gestating in their chest.

Maybe the dangers of alien infestation won’t be a concern for at least ten more years, but right now we already are in the digital space, which packs many threats of its own. To protect your infrastructure from an invasion, allow me to suggest a jump host1 named sshportal.

Distinguishing features

Full independence of users and hosts.

sshportal streamlines their management and makes it easy to have multiple users to multiple hosts. They are decorrelated, as only the bastion knows information about both sides – the end user doesn’t have to know the hosts, he is automatically connected to everything he should have access to. So, for example, if a new developer comes into the company or changes their key, the configuration on each of the target servers don’t have to be changed; if the developer leaves, you can remove their access in seconds.

Access to target features

Kitchen, cargo hold, gym – there are many things that an airlock isn’t. sshportal does its job only at the connection, then it “pipes” everything to the target host. So it supports various advanced features, without having any code related to them. Some tested features are: X11 forwarding, port forwarding, tunneling, ssh-agent, sftp, scp, rsync, git support.

Connecting to servers that don’t support SSH keys

…, but only have username/password, This can be especially true for some hardware like internet router or old computers. Once you configure a remote host with the username/password, sshportal will automatically enter the login information. So, other people won’t have to know the login and password – they will connect as they would to any other hosts, using their personal SSH key to connect to sshportal.

“Roles” that provide various levels of access. A user can have access only to a particular host or host group. Login, ssh access, sftp access, and many other features can be enabled or disabled for specific users.

sshportal is also portable, scalable, provides various kinds of statistics and other neat features.

The “invite” system

sshportal was made to be fluid and easy to install and manage, and adding new users is a task that it simplifies greatly. Let’s take a look at a scenario that often occurs when someone in a company wants to access the server:

This person needs to request access from an administrator, to which the administrator will ask for the “public ssh key” of the user – user provides the key, and the administrator manually adds it on the server. Even if the user provides the key along with the request, the administrator still has to ask the user to test the connection, then the user tries and confirms – or doesn’t, which leads to some more steps. This workflow becomes even more involved when you have a user asking his manager to ask the “admin team” or to “open a ticket”, as every step is asynchronous and non-monitorable.

The “invite” mechanism works the opposite way. Admin can create an “invite code” in advance, with a named account that doesn’t have any public ssh key; the first user connecting to the server with this code will automatically link its key with the account. So, after a user requests the access, the workflow goes like this:

Administrator creates the user account sshportal user create toto@company.com. sshportal returns a one-line instruction, i.e.: ssh portal.company.com -l invite:XXXXX that the administrator can give to the user. A user executes the line, and that’s it.

Also, the administrator can check if the user tried to reconnect or not.

Use cases

Used by educators to provide temporary access to students.

A group of over 6000 people is using it internally to manage access to servers/routers, saving hours on configuration management and not having to share the configuration information.

There are companies who use a jump host to monitor connections at a single point.

A hosting company is using sshportal for its “logging” feature, among the others. As every session is logged and introspectable, they have a detailed history of who performed which action. This company made its own contribution to the project, allowing the support of more than 65.000 sessions in the database.

The project has also received multiple contributions from a security researcher that made a thesis on quantum cryptography. This person uses sshportal in their security-hardened hosting company.

A step beyond

There are more things to say about sshportal, but you could visit the GitHub page to check out the demo for yourself and learn more about its features and inner workings. The project is actively supported and has new features in development, like direct access to docker container. Perhaps there is something you personally would like to add or improve? Welcome aboard!

Footnotes

Jump host1 – also known as “bastion” or “gateway” – is an intermediary host that stands between the user and target hosts. It is exposed to the internet and configured to withstand attacks, while the target host remains in the “protected” network, behind the firewall. For more details, here are the Wiki articles on Bastion host and Demilitarized Zo ne.

Docker Machine driver for Scaleway

We just released the Docker Machine driver for Scaleway; You can now use Machine to create Docker hosts on Scaleway servers.

This tool comes as an alternative method of provisionning Docker engines on Scaleway and won’t replace the Docker Instant-App.

Using Docker-Machine, it is now very easy to create a Swarm cluster in minutes: start-swarm-cluster.sh script (output).

It is now easy to manage a Swarm cluster of hybrid servers (x86_64 + armhf).

It is also very convenient to use a Swarm cluster composed of Raspberry PI using the Hypriot OS and Scaleway servers together.