Manifest¶

A manifest file is a .yaml file describing what external projects are used in this project.

This can be any external repository (git, svn). A section remotes (see Remotes) which contains a list of sources of the projects to download and a section projects: that contains a list of projects to fetch.

manifest:
    version: 0.0

    remotes:
    - name: git-modules
      url-base: http://git.mycompany.local/mycompany/

    projects:
     - name: mymodule
       dst: external/mymodule/

Remotes¶

Remotes are the external repository where the code should be retrieved from.

The remotes: section is not mandatory. If only one remote is added this is assumed to be the default. If multiple remotes are listed default: can be explicitly specified. If multiple remotes are marked as default, the first marked as default is chosen.

manifest:
    version: 0.0

    remotes:
    - name: mycompany-git-modules
      url-base: http://git.mycompany.local/mycompany/
      default: true
    - name: github
      url-base: https://github.com/

Projects¶

Projects are a specific repository or sources to download from a remote location.

In its most basic form a project only has a name:. This would make Dfetch retrieve the mymodule project from the only remote listed (mycompany-git-modules) and place it in a folder mymodule in the same folder as the manifest.

A project name must be unique and each manifest must have at least one project.

manifest:
    version: 0.0

    remotes:
    - name: mycompany-git-modules
      url-base: http://git.mycompany.local/mycompany/

    projects:
     - name: mymodule

Revision/Branch/Tag¶

When no version is provided the latest version of the default branch (e.g. trunk, master) of a project will be chosen. Since we want more control on what project is retrieved the revision:, branch: and tag: attributes can help. Below manifest will download tag v1.13 of mymodule.

The tag: attribute takes priority over revision: and branch:. With revision: a specific commit (git) or revision (svn) is retrieved. For git, revisions must be complete 40 character long SHA-hashes. For git if only a revision is provided and no branch, DFetch cannot determine if there are updates, so it will assume you require that specific version.

manifest:
    version: 0.0

    remotes:
    - name: mycompany-git-modules
      url-base: http://git.mycompany.local/mycompany/

    projects:
     - name: mymodule
       tag: v1.13

     - name: myothermodule
       revision: dcc92d0ab6c4ce022162a23566d44f673251eee4

Note

For svn a standard layout is advised. Meaning the top of the repository has a trunk, branches and tags folder. If this is not the case, you can indicate this by using branch: ' '.

Destination¶

To control where the project is placed, the dst: attribute can be used. Below manifest will place mymodule at the relative path listed by dst: (relative to the manifest location).

manifest:
    version: 0.0

    remotes:
    - name: mycompany-git-modules
      url-base: http://git.mycompany.local/mycompany/

    projects:
     - name: mymodule
       dst: external/mymodule

If no dst: is provided, DFetch will use the project name as relative path. The dst: must be in the same folder or a folder below the manifest and must be unique.

Example: No fetch is done directly into manifest folder

../features/fetch-checks-destination.feature¶

Given the manifest 'dfetch.yaml'
    """
    manifest:
      version: '0.0'

      projects:
        - name: ext/test-repo-tag
          url: https://github.com/dfetch-org/test-repo
          tag: v1
          dst: .

    """
When I run "dfetch update"
Then the output shows
    """
    Dfetch (0.10.0)
      ext/test-repo-tag   : Skipping, path "." is not allowed as destination.
    Destination must be in a valid subfolder. "." is not valid!
    """

Example: Path traversal is not allowed

../features/fetch-checks-destination.feature¶

Given the manifest 'dfetch.yaml'
    """
    manifest:
      version: '0.0'

      projects:
        - name: ext/test-repo-tag
          url: https://github.com/dfetch-org/test-repo
          tag: v1
          dst: ../../some-higher-folder

    """
When I run "dfetch update"
Then the output shows
    """
    Dfetch (0.10.0)
      ext/test-repo-tag   : Skipping, path "../../some-higher-folder" is outside manifest directory tree.
    Destination must be in the manifests folder or a subfolder. "../../some-higher-folder" is outside this tree!
    """

Repo-path¶

When working with remotes by default Dfetch will take the url-base of the remote and concatenate that with the name of the project. Sometimes you want more control of the name, you can use the repo-path: attribute to list it explicitly.

For instance, below example will look for the remote project at <url-base>:<repo-path>, which would be https://github.com/cpputest/cpputest.

manifest:
    version: 0.0

    remotes:
    - name: github
      url-base: https://github.com/

    projects:
    - name: cpputest
      repo-path: cpputest/cpputest

Source¶

In larger projects or mono-repo’s it is often desirable to retrieve only a subfolder of an external project. Dfetch makes this possible through the src: attribute.

For instance if you are only interested in the src folder of cpputest you can limit the checkout to only that folder. Dfetch will retain any license file in the root of the repository.

Example: License is preserved in git repo sparse checkout and cannot be ignored

../features/keep-license-in-project.feature¶

Given the manifest 'dfetch.yaml' in MyProject
    """
    manifest:
        version: 0.0
        projects:
            - name: SomeProjectWithLicense
              url: some-remote-server/SomeProjectWithLicense.git
              src: SomeFolder/
              tag: v1
              ignore:
                  - LICENSE
    """
And a git-repository "SomeProjectWithLicense.git" with the files
    | path                                  |
    | LICENSE                               |
    | SomeFolder/SomeFile.txt               |
    | SomeOtherFolder/SomeOtherFile.txt     |
    | SomeSubFolder/LICENSE                 |
When I run "dfetch update"
Then the output shows
    """
    Dfetch (0.10.0)
      SomeProjectWithLicense: Fetched v1
    """
Then 'MyProject' looks like:
    """
    MyProject/
        SomeProjectWithLicense/
            .dfetch_data.yaml
            LICENSE
            SomeFile.txt
        dfetch.yaml
    """

Example: License is preserved in svn repo sparse checkout and cannot be ignored

../features/keep-license-in-project.feature¶

Given the manifest 'dfetch.yaml' in MyProject
    """
    manifest:
        version: 0.0
        projects:
            - name: SomeProjectWithLicense
              url: some-remote-server/SomeProjectWithLicense
              src: SomeFolder/
              ignore:
                  - LICENSE
    """
And a svn-server "SomeProjectWithLicense" with the files
    | path                                  |
    | LICENSE                               |
    | SomeFolder/SomeFile.txt               |
    | SomeOtherFolder/SomeOtherFile.txt     |
When I run "dfetch update"
Then the output shows
    """
    Dfetch (0.10.0)
      SomeProjectWithLicense: Fetched trunk - 1
    """
Then 'MyProject' looks like:
    """
    MyProject/
        SomeProjectWithLicense/
            .dfetch_data.yaml
            LICENSE
            SomeFile.txt
        dfetch.yaml
    """

Example: A single file is fetched from svn repo

../features/keep-license-in-project.feature¶

Given the manifest 'dfetch.yaml' in MyProject
    """
    manifest:
        version: 0.0
        projects:
            - name: SomeProjectWithLicense
              dst: SomeFile.txt
              url: some-remote-server/SomeProjectWithLicense
              src: SomeFolder/SomeFile.txt
    """
And a svn-server "SomeProjectWithLicense" with the files
    | path                                  |
    | COPYING.txt                           |
    | SomeFolder/SomeFile.txt               |
    | SomeOtherFolder/SomeOtherFile.txt     |
When I run "dfetch update"
Then the output shows
    """
    Dfetch (0.10.0)
      SomeProjectWithLicense: Fetched trunk - 1
    """
Then 'MyProject' looks like:
    """
    MyProject/
        .dfetch_data-SomeFile.txt.yaml
        COPYING.txt
        SomeFile.txt
        dfetch.yaml
    """

manifest:
    version: 0.0

    remotes:
    - name: github
      url-base: https://github.com/

    projects:
    - name: cpputest
      src: src
      repo-path: cpputest/cpputest

It is also possible to use an * to match only certain files with the src tag. The following manifest will only checkout files in folder src with the *.h extension.

manifest:
    version: 0.0

    remotes:
    - name: github
      url-base: https://github.com/

    projects:
    - name: cpputest
      src: src/*.h
      repo-path: cpputest/cpputest

Example: A file pattern is fetched from a repo

../features/fetch-file-pattern-git.feature¶

Given the manifest 'dfetch.yaml' in MyProject
    """
    manifest:
        version: 0.0
        projects:
            - name: SomeProjectWithAnInterestingFile
              url: some-remote-server/SomeProjectWithAnInterestingFile.git
              src: SomeFolder/SomeSubFolder/*.txt
              tag: v1
    """
And a git-repository "SomeProjectWithAnInterestingFile.git" with the files
    | path                                   |
    | SomeFolder/SomeSubFolder/SomeFile.txt  |
    | SomeFolder/SomeSubFolder/OtherFile.txt |
    | SomeFolder/SomeSubFolder/SomeFile.md   |
When I run "dfetch update"
Then the output shows
    """
    Dfetch (0.10.0)
      SomeProjectWithAnInterestingFile: Fetched v1
    """
Then 'MyProject' looks like:
    """
    MyProject/
        SomeProjectWithAnInterestingFile/
            .dfetch_data.yaml
            OtherFile.txt
            SomeFile.txt
        dfetch.yaml
    """

Ignore¶

In larger projects or mono-repo’s it is often desirable to ignore certain files such as Tests Dfetch makes this possible through the ignore: attribute.

For instance if you are not interested in the tests folder of cpputest you can limit the checkout to ignore that folder. Dfetch will retain any license file in the root of the repository.

manifest:
    version: 0.0

    remotes:
    - name: github
      url-base: https://github.com/

    projects:
    - name: cpputest
      ignore:
      - tests
      repo-path: cpputest/cpputest

Note

For projects using src: field, the ignore list will be relative to that folder. And the ignores will be applied after the src: pattern was applied. License files will never be excluded, since you likely shouldn’t be doing that.

Example: A file pattern is fetched from a repo

../features/fetch-with-ignore-git.feature¶

Given the manifest 'dfetch.yaml' in MyProject
    """
    manifest:
        version: 0.0
        projects:
            - name: SomeInterestingProject
              url: some-remote-server/SomeInterestingProject.git
              src: SomeFolder/SomeSubFolder
              ignore:
                 - OtherFile.txt
              tag: v1
    """
When I run "dfetch update"
Then the output shows
    """
    Dfetch (0.10.0)
      SomeInterestingProject: Fetched v1
    """
Then 'MyProject' looks like:
    """
    MyProject/
        SomeInterestingProject/
            .dfetch_data.yaml
            SomeFile.md
            SomeFile.txt
        dfetch.yaml
    """

Example: Combination of directories and a single file can be ignored

../features/fetch-with-ignore-git.feature¶

Given the manifest 'dfetch.yaml' in MyProject
    """
    manifest:
        version: 0.0
        projects:
            - name: SomeInterestingProject
              url: some-remote-server/SomeInterestingProject.git
              ignore:
                 - SomeFolder/SomeOtherSubFolder
                 - SomeFolder/SomeSubFolder/SomeFile.md
              tag: v1
    """
When I run "dfetch update"
Then the output shows
    """
    Dfetch (0.10.0)
      SomeInterestingProject: Fetched v1
    """
Then 'MyProject' looks like:
    """
    MyProject/
        SomeInterestingProject/
            .dfetch_data.yaml
            SomeFolder/
                SomeSubFolder/
                    OtherFile.txt
                    SomeFile.txt
        dfetch.yaml
    """

Example: Ignore overrides the file pattern match in src attribute

../features/fetch-with-ignore-git.feature¶

Given the manifest 'dfetch.yaml' in MyProject
    """
    manifest:
        version: 0.0
        projects:
            - name: SomeInterestingProject
              url: some-remote-server/SomeInterestingProject.git
              src: SomeFolder/SomeSubFolder/*.txt
              ignore:
                 - /SomeNonExistingPath
                 - SomeFile.*
              tag: v1
    """
When I run "dfetch update"
Then the output shows
    """
    Dfetch (0.10.0)
      SomeInterestingProject: Fetched v1
    """
Then 'MyProject' looks like:
    """
    MyProject/
        SomeInterestingProject/
            .dfetch_data.yaml
            OtherFile.txt
        dfetch.yaml
    """

VCS type¶

DFetch does it best to find out what type of version control system (vcs) the remote url is, but sometimes both is possible. For example, GitHub provides an svn and git interface at the same url.

To provide you an option to explicitly state the vcs, the vcs: attribute was introduced. In the below example the same project is fetched as SVN and as Git repository. Dfetch will default to the latest revision from trunk for svn and master from git.

manifest:
    version: 0.0

    remotes:
    - name: github
      url-base: https://github.com/

    projects:
    - name: cpputest
      vcs: git
      repo-path: cpputest/cpputest

    - name: cpputestSVN
      vcs: svn
      repo-path: cpputest/cpputest

Patch¶

DFetch promotes upstreaming changes, but also allows local changes. These changes can be managed with a local patch file. DFetch will apply the patch file every time a new upstream version is fetched. The patch file can be specified with the patch: attribute.

manifest:
    version: 0.0

    remotes:
    - name: github
      url-base: https://github.com/

    projects:
    - name: cpputest
      vcs: git
      repo-path: cpputest/cpputest
      patch: local_changes.patch

The patch can be generated using the Dfetch Diff command. Alternately the patch can be generated manually as such. Note that it should be relative.

# For git repo's
git diff --relative=path/to/project HEAD > my_patch.patch

# For svn repo's
svn diff -r HEAD path/to/my_project > my_patch.patch

For more details see the git-diff or svn-diff documentation.

Example: A patch file is generated

../features/diff-in-git.feature¶

Given "SomeProject/README.md" in MyProject is changed and committed with
    """
    An important sentence for the README!
    """
When I run "dfetch diff SomeProject"
Then the patch file 'MyProject/SomeProject.patch' is generated
    """
    diff --git a/README.md b/README.md
    index 1e65bd6..faa3b21 100644
    --- a/README.md
    +++ b/README.md
    @@ -1 +1,2 @@
    Generated file for SomeProject.git
    +An important sentence for the README!
    """

Example: No change is present

../features/diff-in-git.feature¶

When I run "dfetch diff SomeProject"
Then the output shows
"""
Dfetch (0.10.0)
  SomeProject         : No diffs found since 59efb91396fd369eb113b43382783294dc8ed6d2
"""

Example: Diff is generated on uncommitted changes

../features/diff-in-git.feature¶

Given "SomeProject/README.md" in MyProject is changed with
    """
    An important sentence for the README!
    """
When I run "dfetch diff SomeProject"
Then the patch file 'MyProject/SomeProject.patch' is generated
    """
    diff --git a/README.md b/README.md
    index 1e65bd6..faa3b21 100644
    --- a/README.md
    +++ b/README.md
    @@ -1 +1,2 @@
    Generated file for SomeProject.git
    +An important sentence for the README!
    """

Manifest¶

Remotes¶

Projects¶

Revision/Branch/Tag¶

Destination¶

Repo-path¶

Source¶

Ignore¶

VCS type¶

Patch¶

Dfetch

Navigation

Related Topics