Shared file and resources methods

January 19, 2021

Our development team is constantly improving the RUDDER software. They have to be at the cutting edge of technology. So they give you tips and feedbacks on the codes and techniques they use when they discover something interesting.

Managing configuration files is one of the, if not the, most crucial part of IT infrastructure management and the most efficient way to achieve it is by sharing resources (files, templates) to all your nodes. But, ensuring that the correct files are deployed on hundreds, thousands of machines is a complex and time consuming task (some would say a nightmare), which means that you need tools that provide a reliable, secure and performant way to provide a total control of this process. And this is where Rudder comes into play.

Rudder provides a central server to share your resources through policies you defined in our web UI and ensure that they were correctly distributed to all your nodes with our compliance system. In Rudder latest releases, file distribution has been significantly improved: this article will give you an overview of the current options you have to distribute files with it.

Introduction to file sharing

Since Rudder 6.1, there are 3 options to share files to your nodes:

Downloading from Rudder server shared folder: Rudder server acts as file repository that nodes can fetch
Linked ressources with configuration policies: define your configuration and share files packaged with it in one go
Share files between nodes: Send files from a Node to other Nodes of your IT

We’ll cover more in detail, in the next section, some use cases related to each of the following options.

Downloading from Rudder server shared folder

Easiest and oldest way to share resources is to share a file through Rudder server. Files needs to be copied within the /var/rudder/configuration-repository/shared-files (which is a git repository so you can also track an history of you changes here, or even synch git submodules) Once your files have been dropped in the right place, you have two options to build your download policy:

Either going through the built in technique in the directive section which is called ‘File Download (Rudder server)’.

This technique includes few parameters to fill and requires that resources that need to be shared to your nodes to be added to the configuration-repository
During the agent execution, it ensures that all files are updated within the nodes.
Usually this method is dedicated to large files transfer.

You can also create your own policies including file sharing using the technique editor and the following methods: File copy from rudder shared folder File from remote source

‘File Download (Rudder server)’ technique

As we mentioned above, you can simply use the first option via ‘File Download (Rudder server)’ technique.

In the figure below, let’s suppose that you want to copy/synchronize the directory ‘my-app/conf-files’ with your nodes. First, there is a bunch of options that you can use to specify what need to be copied, such as including/excluding files from the copy, defining the recursion level of the copy, etc.

Next, you can define options such as permission and ownership of files once they are copied in the right place.

Finally, you can add a command that will be executed after the download, if any changes occured.

Once the configuration is OK and the corresponding policy is deployed on agent, during agent execution the whole directory ‘my-app/conf-files’ will be downloaded by the nodes as part of the files transferring process. Then it will be maintained and synchronized whenever there will be changes in its content on the shared-files directory on the server. Of course, if files were changed on the node, they will also be replaced by a fresh copy from the server.

Also you can optionally add a command that will be executed following any changes in the resources content.

Using methods from the technique editor

When building your own technique, you may want to share files from Rudder server to your nodes. The main interest here is that you have more flexibility than the built-in technique and you can choose what to do with the file downloaded from Rudder server, like applying a templating engine on the file to it final path.

There are two methods that can be used:

File copy from Rudder shared folder
File from remote source

File copy from Rudder shared folder

This method downloads files from Rudder server ‘shared-files’ folder, like the built-in technique.

File from remote source

This generic method (and recursive variant “File from remote source recursion”) allows file transfer from a source path on the server to a target path on the node. note that source directory should be allowed on the root server, and only the shared-files is available by default

Linked ressources with configuration policies

Introduced in 6.1, You can embed files directly within your technique when editing them in the technique editor. This provides a strong link between the resources and the technique and has some advantages over previous download methods.

Resources are historised with your technique and are committed within Rudder git repository allowing you to track changes over time, checking for changes in both logic and files together.
Resources are downloaded when Rudder updates its policies ensuring that the agent will run with a version of the resource consistent with the policy, whereas previous download methods check and download files at agent execution could fail and lead to inconsistencies between a policy and its resources.
Resources folder path can be accessed directly using the ${ressources_dir} variable within the technique in the technique editor.

For example, let’s consider that you have an ansible playbook that you want to launch to configure some network appliance you can only access through ssh but ensuring that you get ansible feedback and keep a compliance trace in Rudder

Go to your technique editor and create a new technique and define your resource in the “Resources” tab

File sharing between nodes

The final option covers a different need than the previous methods: sharing files between nodes. This allows to sync state between nodes by setting a flag file to share from one node, or maybe share a configuration file or a build artifact that has finished to build from a node to another.

This needs to be done in two techniques within the technique editor: one should call “Share file to node”, where you state which file to share to which node:

The other method “Sharedfile from node” should be filled with the source id and where to put the file on the target node:

Once the methods are applied on nodes, the source will send back the file to Rudder server which will make it available for the target nodes which will then download the file.

Conclusion

So this concludes our tour of the different options of sharing files with Rudder, which offers various styles so you can adapt to any of your use cases. We went from highly customizable techniques (with technique editor), to ready to use but more limited built in directives through specific use cases sharing files between nodes. Mixed with other features, compliance, templating and node properties, Rudder is a great tool to ensure that your configuration files are correctly shared with personalized values from your nodes. Don’t hesitate to comment here or contact us about a use case, we may find out together how to achieve it with Rudder, or we can look on how to add this within Rudder in a new Version

Ikram Bejaoui

This section allow you to modify the background image of the post template’s header.