DAQ development

Structuring of version control
Committing to an already existing or a new branch
- Introduction
- Workflow

WIP

Scraps and leftovers from a README previously distributed with the ausadaq super project

Has to be repurposed for this wiki page which is now separate from the other DAQ-related wiki pages

Structuring of version control

The default branch of this git project is the master branch. This branch contains a functioning default setup from which other functioning setups for use at IFA and around the globe should ideally only differ in a few configuration files, which describe the differing NIM modules, VME modules and detectors being utilised in a given setup.

This project does not abide by the commandments of Git Flow or Github Flow, as this would lead to unmanageable “spaghetti branching” within a few years – see the figure in the link for a mild example of this, and do NOT be tempted by whatever the article might be advocating!

A prospective user is first expected to clone this repository to the controller of the DAQ system, which is to be set up,

git clone --recurse-submodules /path/to/ausadaq.git

The user then navigates into the resulting directory and checks out a relevant branch – named after a given experimental setup/experiment, we refer to the branch as exp – which initiates the fetching of the sub projects contained in this super project

cd ausadaq
git checkout --recurse-submodules <exp>

The contents of the git super project (on branch master) could look as follows

q@bumblebee:~/daq> ls -lah
total 52K
drwxr-xr-x  6 q q 4,0K Jan 24 10:51 .
drwxr-xr-x  3 q q 4,0K Jan 24 10:51 ..
drwxr-xr-x  9 q q 4,0K Jan 24 10:51 board
drwxr-xr-x 13 q q 4,0K Jan 24 10:51 controller
-rwxr-xr-x  1 q q 7,1K Jan 24 10:51 daqenv
drwxr-xr-x  9 q q 4,0K Jan 24 10:53 .git
-rw-r--r--  1 q q  251 Jan 24 10:51 .gitignore
-rw-r--r--  1 q q 1,1K Jan 24 10:51 .gitmodules
-rwxr-xr-x  1 q q 5,1K Jan 24 10:51 prepare_software.sh
-rw-r--r--  1 q q 4,0K Jan 24 10:51 README.md
drwxr-xr-x  2 q q 4,0K Jan 24 10:51 util
q@bumblebee:~/daq> git submodule status
 eb503450238626ce25d13fb722b4bc9af2ead4b2 board/software/f_user (base1)
 d5db6f4b54fec95eaf39429b35b07b40b2d6c294 board/software/f_user_conf (heads/master)
 fde3edfbf2ab54f4fad23f3fc47b3cefdfe0b30c board/software/htools (base1)
 8ab5da35000a75db0f22e5aa1436a06cc1defdae board/software/nurdlib (base1)
 24551b654d3c32d47ace70cf17c41c3cf55b0062 board/software/trloii (base1)
 638d6524c6fc6df444e1976b6774479f13ec58df controller/software/ausalib (base1)
 5c8ce1bf95d58f9af2c9be0ff10c2edcec89aa34 controller/software/drasi (base1)
 892f9a83873e20e55ae47c20a40b479f3d5ecd09 controller/software/go4cesb (heads/master)
 e81dab93863e8499c4ca5ce24f9af380ed4c368a controller/software/pfeiffer (base1)
 fbf0467a0cba641f3522538bb8123eceaa083c81 controller/software/sorter (base1)
 ca7669d4d2d70a26781e12613e3d97aa429ba2ab controller/software/ucesb (base1)
 259a7b888a2a96c67fbb2936df0d9c110c668b73 controller/software/unpacker (heads/master)
q@bumblebee:~/daq>

Each branch in the super project repository unambigously specifies the versions of software of the sub projects which have previously been set up to define a functioning DAQ for the experimental setup which gave the relevant branch its name; configuration files which helped define the functioning DAQ are also included.

When setting up a DAQ for a new experiment or when modifying an already existing one, the user will have checked out an already functioning configuration. The user should commit and push the changes they make either to a new branch, defining a new experiment, or to an already existing branch, updating an already existing experiment, only when the DAQ is functional under the new configuration.

This git repository is to be structured with as little “spaghetti branching” as possible: The project has a master branch, and from it all other branches stick out and run alongside it, never to merge with it or any other branches ever again. Whoever has to set up a drasi-TRLOII-Nurdlib-ucesb-Go4-based VME DAQ next, should have as easy a time as possible, getting an overview of the project structure. Do NOT push your own development branches here!

(…)

Committing to an already existing or a new branch

Introduction

The git super project contains two types of git sub projects:

Stable repositories – these are very rarely modified and will tend to be checked out on the same git tags across many different setups
Unstable repositories – because each of our experiments varies in which modules, detectors, etc. they use, and because these repositories reflect this in their configurations, these are ever-changing

Our stable repositories only have their own master branch. Functioning combinations of these repos are tagged baseX where X is an unsigned integer. On the rare occasion that a major update is carried out on one of the stable repositories, X is to be incremented on all stable repositories on the master branch. Needless to say, it should be tested that all other stable and unstable repositories still function under the new update of the one stable repository in question.

Our unstable repositories, on the other hand, have as many branches as we have/have had experiments (counting from 2023). Functioning combinations of these repos are also tagged, but the naming here is not so strict. If it makes sense to have several tags on a given experiment’s branch, this can be done, for example to have some sort of versioning – otherwise, a single tag on a given branch could just have the same name as the branch itself.

Finally, the git super project itself has as many branches as we have/have had experiments (counting from 2023), and it has a master branch, which defines whatever makes sense to be the working default setup of our group at a given moment. The git super project keeps track of which combinations of git branches and tags of the various sub projects belong to each of our experiments.

Workflow

Having made additions, changes, improvements - you name it! - to one or more submodules of this project or to this super project itself, here is how we specifically commit and push changes to the repositories.

Note: Git is very eager to create lots of branches and merge only whenever it renders things the most confusing; it quickly becomes a giant mess! Our approach – in order to make the evolution of our setups as easy to follow as possible, whilst having reproducible setups – is the following:

By default, be in a detached HEAD state on a specific git tag which describes (parts of) a functioning setup; the git tag is attached to a specific git branch, and together they help describe the evolution of the setup related to a given experiment
During preparations for a new setup/experiment or during an actual experiment, small modifications in the various (sub)modules of the DAQ software/configuration will naturally be carried out
When the modifications seem to have settled momentarily, the modifications are stashed – i.e. the changes are saved in a temporary location known only to Git and the repository/repositories are locally brought back to their default detached HEAD state without modifications
This allows for the local repo/repos to be checked out on the relevant git branch, clearing the detached HEAD state
The stashed changes are then brought back in, overriding the contents on the git branch locally, and these changes are added and commited to the local repo/repos before they are pushed to the remote repo/repos
Next, either the original git tag is removed from the older commit onto this newer commit or a new git tag is applied to this newer commit; the latter case applies if it makes sense to have some sort of versioning for a given setup
Finally, check out the local repo/repos onto this most recent git tag, returning the repo/repos to their default detached HEAD state

Note (refer to the second-to-last bullet point above) that the tip of a git branch is never without a tag!

We demonstrate how this all works in practice by example.

Workflow by example: Stable repository

(Remember that we have two kinds of git sub projects: stable and unstable repositories. This example deals with a stable repository.)

Imagine that we have made some changes to one of our stable repositories which is currently detached on the tag base1 on the master branch. The name of our remote is origin, and Git reports that we have unstaged changes and untracked files:

q@bumblebee:/path/to/repo$ git remote -v
origin	<remote url> (fetch)
origin	<remote url> (push)
q@bumblebee:/path/to/repo$ git status
HEAD detached at base1
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   file_with_modifications

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	new_file

no changes added to commit (use "git add" and/or "git commit -a")
q@bumblebee:/path/to/repo$

First we need to ensure that whatever changes we make will not clash with someone else’s by pulling any remote modifications to our local version of the master branch:

q@bumblebee:~/base/board/software/htools$ git pull origin master
From <remote url>
 * branch            master     -> FETCH_HEAD
Already up-to-date.
q@bumblebee:/path/to/repo$ git status
HEAD detached at base1
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   file_with_modifications

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	new_file

no changes added to commit (use "git add" and/or "git commit -a")
q@bumblebee:/path/to/repo$

We stash our changes and confirm that we are still in a detached HEAD state:

q@bumblebee:/path/to/repo$ git stash
Saved working directory and index state WIP on (no branch): <commit hash> <commit message>
HEAD is now at <commit hash> <commit message>
q@bumblebee:/path/to/repo$ git status
HEAD detached at base1
nothing to commit, working directory clean
q@bumblebee:/path/to/repo$

Since the working directory is now clean, we can checkout the master branch and pop back in our stashed changes:

q@bumblebee:/path/to/repo$ git checkout master
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
q@bumblebee:/path/to/repo$ git stash pop
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

	new file:   new_file

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   file_with_modifications

Dropped refs/stash@{0} (<long commit hash>)
q@bumblebee:/path/to/repo$

Note that we are no longer in a detached HEAD state. This means that whatever local changes we stage and commit, we can also push to the tip of the current branch to the remote repository. We stage our changes:

q@bumblebee:/path/to/repo$ git add new_file file_with_modifications 
q@bumblebee:/path/to/repo$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

	modified:   file_with_modifications
	new file:   new_file
q@bumblebee:/path/to/repo$

We then commit our changes and check the status reported by Git:

q@bumblebee:/path/to/repo$ git commit -m "<description of additions/modifications>"
[master <commit hash>] <description of additions/modifications>
 X files changed, Y insertions(+)
(...)
q@bumblebee:/path/to/repo$ git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)
q@bumblebee:/path/to/repo$

We go ahead and push our changes to the remote repository:

q@bumblebee:/path/to/repo$ git push
Total 0 (delta 0), reused 0 (delta 0)
To <remote url>
   <commit hash 1>..<commit hash 2>  master -> master
q@bumblebee:/path/to/repo$

Now, we have to apply a tag to the new tip of the master branch. Let us imagine that the changes to the repository here do not affect any other repository in any way and that the changes are so minor that it does not make sense to increment the tag name from base1 to base2. What we want to do, then, is delete the tag base1 from the older commit, such that we can apply it to this newer commit.

In order, we list the tags available locally and then we list the tags available at the remote repository:

q@bumblebee:/path/to/repo$ git tag -l
base1
q@bumblebee:/path/to/repo$ git ls-remote
From <remote url>
<long commit hash A>				HEAD
<long commit hash A>				refs/heads/master
<long commit hash B>				refs/tags/base1
q@bumblebee:/path/to/repo$

We delete the tag base1 locally and confirm that it has indeed been deleted:

q@bumblebee:/path/to/repo$ git tag -d base1 
Deleted tag 'base1' (was <commit hash>)
q@bumblebee:/path/to/repo$ git tag -l
q@bumblebee:/path/to/repo$

We then delete the tag base1 remotely by “pushing” to the remote, and we confirm that the tag has indeed been deleted:

q@bumblebee:/path/to/repo$ git push -v --delete origin base1
Pushing to <remote url>
To <remote url>
 - [deleted]         base1
q@bumblebee:/path/to/repo$ git ls-remote
From <remote url>
<long commit hash A>				HEAD
<long commit hash A>				refs/heads/master
q@bumblebee:/path/to/repo$

We then tag our latest commit:

q@bumblebee:/path/to/repo$ git tag base1
q@bumblebee:/path/to/repo$ git tag -l
base1
q@bumblebee:/path/to/repo$

Finally, we apply this newer base1 tag to the tip of the remote:

q@bumblebee:/path/to/repo$ git push --tags --prune
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 309 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
To <remote url>
 * [new tag]         base1 -> base1
q@bumblebee:/path/to/repo$ git ls-remote
From <remote url>
<long commit hash A>				HEAD
<long commit hash A>				refs/heads/master
<long commit hash A>				refs/tags/base1
q@bumblebee:/path/to/repo$

We see that we are still on branch master, with nothing more to do, and we then checkout to the newly created base1 tag, returning to our default detached HEAD state:

q@bumblebee:/path/to/repo$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
q@bumblebee:/path/to/repo$ git checkout base1
Note: checking out 'base1'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at <commit hash>... <commit message>
q@bumblebee:/path/to/repo$ git status
HEAD detached at base1
nothing to commit, working directory clean
q@bumblebee:/path/to/repo$

AN EXAMPLE FROM AN UNSTABLE REPO GOES HERE

HOW TO UPDATE THE SUPER PROJECT (E.G. AFTER SUB PROJECTS HAVE BEEN UPDATED) GOES HERE

SOMETHING ABOT REBASING GOES HERE