Manifest¶

A manifest file is a .yaml file describing what external projects are used in this project.

This can be any external repository (git, svn). A section remotes (see Remotes) which contains a list of sources of the projects to download and a section projects: that contains a list of projects to fetch.

manifest:
    version: 0.0

    remotes:
    - name: git-modules
      url-base: http://git.mycompany.local/mycompany/

    projects:
     - name: mymodule
       dst: external/mymodule/

Remotes¶

Remotes are the external repository where the code should be retrieved from.

The remotes: section is not mandatory. If only one remote is added this is assumed to be the default. If multiple remotes are listed default: can be explicitly specified. If multiple remotes are marked as default, the first marked as default is chosen.

manifest:
    version: 0.0

    remotes:
    - name: mycompany-git-modules
      url-base: http://git.mycompany.local/mycompany/
      default: true
    - name: github
      url-base: https://github.com/

Projects¶

Projects are a specific repository or sources to download from a remote location.

In its most basic form a project only has a name:. This would make Dfetch retrieve the mymodule project from the only remote listed (mycompany-git-modules) and place it in a folder mymodule in the same folder as the manifest.

A project name must be unique and each manifest must have at least one project.

manifest:
    version: 0.0

    remotes:
    - name: mycompany-git-modules
      url-base: http://git.mycompany.local/mycompany/

    projects:
     - name: mymodule

Revision/Branch/Tag¶

When no version is provided the latest version of the default branch (e.g. trunk, master, main) of a project will be chosen. Since we want more control on what project is retrieved the revision:, branch: and tag: attributes can help. Below manifest will download tag v1.13 of mymodule.

The tag: attribute takes priority over revision: and branch:. With revision: a specific commit (git) or revision (svn) is retrieved. For git, revisions must be complete 40 character long SHA-hashes. For git if only a revision is provided and no branch, DFetch cannot determine if there are updates, so it will assume you require that specific version.

manifest:
    version: 0.0

    remotes:
    - name: mycompany-git-modules
      url-base: http://git.mycompany.local/mycompany/

    projects:
     - name: mymodule
       tag: v1.13

     - name: myothermodule
       revision: dcc92d0ab6c4ce022162a23566d44f673251eee4

Note

For svn a standard layout is advised. Meaning the top of the repository has a trunk, branches and tags folder. If this is not the case, you can indicate this by using branch: ' '.

Destination¶

To control where the project is placed, the dst: attribute can be used. Below manifest will place mymodule at the relative path listed by dst: (relative to the manifest location).

manifest:
    version: 0.0

    remotes:
    - name: mycompany-git-modules
      url-base: http://git.mycompany.local/mycompany/

    projects:
     - name: mymodule
       dst: external/mymodule

If no dst: is provided, DFetch will use the project name as relative path. The dst: must be in the same folder or a folder below the manifest and must be unique.

Example: No fetch is done directly into manifest folder

Scenario: No fetch is done directly into manifest folder
    Given the manifest 'dfetch.yaml'
        """
        manifest:
          version: '0.0'

          projects:
            - name: ext/test-repo-tag
              url: https://github.com/dfetch-org/test-repo
              tag: v1
              dst: .

        """
    When I run "dfetch update"
    Then the output shows
        """
        Dfetch (0.14.3)
          ext/test-repo-tag:
          > Skipping, path "." is not allowed as destination.
        """

Example: Path traversal is not allowed

Scenario: Path traversal is not allowed
    Given the manifest 'dfetch.yaml'
        """
        manifest:
          version: '0.0'

          projects:
            - name: ext/test-repo-tag
              url: https://github.com/dfetch-org/test-repo
              tag: v1
              dst: ../../some-higher-folder

        """
    When I run "dfetch update"
    Then the output shows
        """
        Dfetch (0.14.3)
          ext/test-repo-tag:
          > Skipping, path "../../some-higher-folder" is outside manifest directory tree.
        """

Repo-path¶

When working with remotes by default Dfetch will take the url-base of the remote and concatenate that with the name of the project. Sometimes you want more control of the name, you can use the repo-path: attribute to list it explicitly.

For instance, below example will look for the remote project at <url-base>:<repo-path>, which would be https://github.com/cpputest/cpputest.

manifest:
    version: 0.0

    remotes:
    - name: github
      url-base: https://github.com/

    projects:
    - name: cpputest
      repo-path: cpputest/cpputest

Source¶

In larger projects or mono-repo’s it is often desirable to retrieve only a subfolder of an external project. Dfetch makes this possible through the src: attribute.

For instance if you are only interested in the src folder of cpputest you can limit the checkout to only that folder. Dfetch will retain any license file in the root of the repository.

Example: License is preserved in git repo sparse checkout and cannot be ignored

Scenario: License is preserved in git repo sparse checkout and cannot be ignored
    Given the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
            version: 0.0
            projects:
                - name: SomeProjectWithLicense
                  url: some-remote-server/SomeProjectWithLicense.git
                  src: SomeFolder/
                  tag: v1
                  ignore:
                      - LICENSE
        """
    And a git-repository "SomeProjectWithLicense.git" with the files
        | path                                  |
        | LICENSE                               |
        | SomeFolder/SomeFile.txt               |
        | SomeOtherFolder/SomeOtherFile.txt     |
        | SomeSubFolder/LICENSE                 |
    When I run "dfetch update"
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeProjectWithLicense/
                .dfetch_data.yaml
                LICENSE
                SomeFile.txt
            dfetch.yaml
        """

Example: License is preserved in svn repo sparse checkout and cannot be ignored

Scenario: License is preserved in svn repo sparse checkout and cannot be ignored
    Given the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
            version: 0.0
            projects:
                - name: SomeProjectWithLicense
                  url: some-remote-server/SomeProjectWithLicense
                  src: SomeFolder/
                  ignore:
                      - LICENSE
        """
    And a svn-server "SomeProjectWithLicense" with the files
        | path                                  |
        | LICENSE                               |
        | SomeFolder/SomeFile.txt               |
        | SomeOtherFolder/SomeOtherFile.txt     |
    When I run "dfetch update"
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeProjectWithLicense/
                .dfetch_data.yaml
                LICENSE
                SomeFile.txt
            dfetch.yaml
        """

Example: A single file is fetched from svn repo

Scenario: A single file is fetched from svn repo
    Given the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
            version: 0.0
            projects:
                - name: SomeProjectWithLicense
                  dst: SomeFile.txt
                  url: some-remote-server/SomeProjectWithLicense
                  src: SomeFolder/SomeFile.txt
        """
    And a svn-server "SomeProjectWithLicense" with the files
        | path                                  |
        | COPYING.txt                           |
        | SomeFolder/SomeFile.txt               |
        | SomeOtherFolder/SomeOtherFile.txt     |
    When I run "dfetch update"
    Then 'MyProject' looks like:
        """
        MyProject/
            .dfetch_data-SomeFile.txt.yaml
            COPYING.txt
            SomeFile.txt
            dfetch.yaml
        """

manifest:
    version: 0.0

    remotes:
    - name: github
      url-base: https://github.com/

    projects:
    - name: cpputest
      src: src
      repo-path: cpputest/cpputest

It is also possible to use an * to match only certain files with the src tag. The following manifest will only checkout files in folder src with the *.h extension.

manifest:
    version: 0.0

    remotes:
    - name: github
      url-base: https://github.com/

    projects:
    - name: cpputest
      src: src/*.h
      repo-path: cpputest/cpputest

Example: A file pattern is fetched from a repo

Scenario: A file pattern is fetched from a repo
    Given the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
            version: 0.0
            projects:
                - name: SomeProjectWithAnInterestingFile
                  url: some-remote-server/SomeProjectWithAnInterestingFile.git
                  src: SomeFolder/SomeSubFolder/*.txt
                  tag: v1
        """
    And a git-repository "SomeProjectWithAnInterestingFile.git" with the files
        | path                                   |
        | SomeFolder/SomeSubFolder/SomeFile.txt  |
        | SomeFolder/SomeSubFolder/OtherFile.txt |
        | SomeFolder/SomeSubFolder/SomeFile.md   |
        | SomeFolder/Unrelated.txt               |
        | AlsoUnrelated.txt                      |
    When I run "dfetch update"
    Then the output shows
        """
        Dfetch (0.14.3)
          SomeProjectWithAnInterestingFile:
          > Fetched v1
        """
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeProjectWithAnInterestingFile/
                .dfetch_data.yaml
                OtherFile.txt
                SomeFile.txt
            dfetch.yaml
        """

Example: A file pattern matches two files in different subfolders

Scenario: A file pattern matches two files in different subfolders
    Given the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
            version: 0.0
            projects:
                - name: SomeProjectWithAnInterestingFile
                  url: some-remote-server/SomeProjectWithAnInterestingFile.git
                  src: SomeFolder/Some*
                  tag: v1
        """
    And a git-repository "SomeProjectWithAnInterestingFile.git" with the files
        | path                                       |
        | SomeFolder/SomeSubFolder/SomeFile.txt      |
        | SomeFolder/SomeOtherSubFolder/SomeFile.txt |
    When I run "dfetch update"
    Then the output shows
        """
        Dfetch (0.14.3)
        The 'src:' filter 'SomeFolder/Some*' matches multiple directories from 'some-remote-server/SomeProjectWithAnInterestingFile.git'. Only considering files in 'SomeFolder/SomeOtherSubFolder'.
          SomeProjectWithAnInterestingFile:
          > Fetched v1
        """
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeProjectWithAnInterestingFile/
                .dfetch_data.yaml
                SomeFile.txt
            dfetch.yaml
        """

Example: A file pattern is fetched from a repo

Scenario: A file pattern is fetched from a repo
    Given the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
            version: 0.0
            projects:
                - name: SomeProjectWithAnInterestingFile
                  url: some-remote-server/SomeProjectWithAnInterestingFile
                  src: SomeFolder/SomeSubFolder/*.txt
        """
    And a svn-server "SomeProjectWithAnInterestingFile" with the files
        | path                                   |
        | SomeFolder/SomeSubFolder/SomeFile.txt  |
        | SomeFolder/SomeSubFolder/OtherFile.txt |
        | SomeFolder/SomeSubFolder/SomeFile.md   |
    When I run "dfetch update"
    Then the output shows
        """
        Dfetch (0.14.3)
          SomeProjectWithAnInterestingFile:
          > Fetched trunk - 1
        """
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeProjectWithAnInterestingFile/
                .dfetch_data.yaml
                OtherFile.txt
                SomeFile.txt
            dfetch.yaml
        """

Ignore¶

In larger projects or mono-repo’s it is often desirable to ignore certain files such as Tests Dfetch makes this possible through the ignore: attribute.

For instance if you are not interested in the tests folder of cpputest you can limit the checkout to ignore that folder. Dfetch will retain any license file in the root of the repository.

manifest:
    version: 0.0

    remotes:
    - name: github
      url-base: https://github.com/

    projects:
    - name: cpputest
      ignore:
      - tests
      repo-path: cpputest/cpputest

Note

For projects using src: field, the ignore list will be relative to that folder. And the ignores will be applied after the src: pattern was applied. License files will never be excluded, since you likely shouldn’t be doing that.

Example: A file pattern is fetched from a repo

Scenario: A file pattern is fetched from a repo
    Given the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
            version: 0.0
            projects:
                - name: SomeInterestingProject
                  url: some-remote-server/SomeInterestingProject.git
                  src: SomeFolder/SomeSubFolder
                  ignore:
                     - OtherFile.txt
                  tag: v1
        """
    When I run "dfetch update"
    Then the output shows
        """
        Dfetch (0.14.3)
          SomeInterestingProject:
          > Fetched v1
        """
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeInterestingProject/
                .dfetch_data.yaml
                SomeFile.md
                SomeFile.txt
            dfetch.yaml
        """

Example: Combination of directories and a single file can be ignored

Scenario: Combination of directories and a single file can be ignored
    Given the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
            version: 0.0
            projects:
                - name: SomeInterestingProject
                  url: some-remote-server/SomeInterestingProject.git
                  ignore:
                     - SomeFolder/SomeOtherSubFolder
                     - SomeFolder/SomeSubFolder/SomeFile.md
                  tag: v1
        """
    When I run "dfetch update"
    Then the output shows
        """
        Dfetch (0.14.3)
          SomeInterestingProject:
          > Fetched v1
        """
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeInterestingProject/
                .dfetch_data.yaml
                SomeFolder/
                    SomeSubFolder/
                        OtherFile.txt
                        SomeFile.txt
            dfetch.yaml
        """

Example: Ignore overrides the file pattern match in src attribute

Scenario: Ignore overrides the file pattern match in src attribute
    Given the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
            version: 0.0
            projects:
                - name: SomeInterestingProject
                  url: some-remote-server/SomeInterestingProject.git
                  src: SomeFolder/SomeSubFolder/*.txt
                  ignore:
                     - /SomeNonExistingPath
                     - SomeFile.*
                  tag: v1
        """
    When I run "dfetch update"
    Then the output shows
        """
        Dfetch (0.14.3)
          SomeInterestingProject:
          > Fetched v1
        """
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeInterestingProject/
                .dfetch_data.yaml
                OtherFile.txt
            dfetch.yaml
        """

Example: A file pattern is fetched from a repo

Scenario: A file pattern is fetched from a repo
    Given the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
            version: 0.0
            projects:
                - name: SomeInterestingProject
                  url: some-remote-server/SomeInterestingProject
                  src: SomeFolder/SomeSubFolder
                  ignore:
                     - OtherFile.txt
        """
    When I run "dfetch update"
    Then the output shows
        """
        Dfetch (0.14.3)
          SomeInterestingProject:
          > Fetched trunk - 1
        """
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeInterestingProject/
                .dfetch_data.yaml
                SomeFile.md
                SomeFile.txt
            dfetch.yaml
        """

Example: Combination of directories and a single file can be ignored

Scenario: Combination of directories and a single file can be ignored
    Given the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
            version: 0.0
            projects:
                - name: SomeInterestingProject
                  url: some-remote-server/SomeInterestingProject
                  ignore:
                     - SomeFolder/SomeOtherSubFolder
                     - SomeFolder/SomeSubFolder/SomeFile.md
        """
    When I run "dfetch update"
    Then the output shows
        """
        Dfetch (0.14.3)
          SomeInterestingProject:
          > Fetched trunk - 1
        """
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeInterestingProject/
                .dfetch_data.yaml
                SomeFolder/
                    SomeSubFolder/
                        OtherFile.txt
                        SomeFile.txt
            dfetch.yaml
        """

Example: Ignore overrides the file pattern match in src attribute

Scenario: Ignore overrides the file pattern match in src attribute
    Given the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
            version: 0.0
            projects:
                - name: SomeInterestingProject
                  url: some-remote-server/SomeInterestingProject
                  src: SomeFolder/SomeSubFolder/*.txt
                  ignore:
                     - /SomeNonExistingPath
                     - SomeFile.*
        """
    When I run "dfetch update"
    Then the output shows
        """
        Dfetch (0.14.3)
          SomeInterestingProject:
          > Fetched trunk - 1
        """
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeInterestingProject/
                .dfetch_data.yaml
                OtherFile.txt
            dfetch.yaml
        """

VCS type¶

DFetch does its best to find out what type of version control system (vcs) the remote url is, for instance by trying a simple call to the remote repository. But sometimes both are possible, for example, in the past GitHub provided an svn and git interface at the same url.

To provide you an option to explicitly state the vcs, the vcs: attribute was introduced. In the below example the same project is fetched as SVN and as Git repository. Dfetch will default to the latest revision from trunk for svn and master from git.

manifest:
    version: 0.0

    remotes:
    - name: github
      url-base: https://github.com/

    projects:
    - name: cpputest
      vcs: git
      repo-path: cpputest/cpputest

    - name: cpputestSVN
      vcs: svn
      repo-path: cpputest/cpputest

Archive¶

Projects distributed as .tar.gz, .tgz, .tar.bz2, .tar.xz or .zip archive files can be fetched using vcs: archive. DFetch downloads the archive from the url: and extracts it to the destination directory, stripping the top-level directory if present.

The src: and ignore: attributes work the same way as for git/SVN projects.

manifest:
    version: 0.0

    projects:
    - name: my-library
      vcs: archive
      url: https://example.com/releases/my-library-1.0.tar.gz

Integrity verification¶

Use the integrity: block to verify the integrity of the downloaded archive. Currently the hash: sub-field is supported (format <algorithm>:<hex-digest>); only sha256 is recognised today, but the block is designed to grow to support detached signature verification via sig: (signature URL) and sig-key: (signing-key URL or fingerprint) in the future.

manifest:
    version: 0.0

    projects:
    - name: my-library
      vcs: archive
      url: https://example.com/releases/my-library-1.0.tar.gz
      integrity:
        hash: sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Run dfetch freeze after an initial dfetch update to add the sha256 hash to the manifest automatically.

Example: Tar.gz archive project is fetched

Scenario: Tar.gz archive project is fetched
    Given an archive "SomeProject.tar.gz" with the files
        | path       |
        | README.md  |
        | src/main.c |
    And the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
          version: '0.0'
          projects:
            - name: SomeProject
              url: some-remote-server/SomeProject.tar.gz
              vcs: archive
        """
    When I run "dfetch update" in MyProject
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeProject/
                .dfetch_data.yaml
                README.md
                src/
                    main.c
            dfetch.yaml
        """

Example: Zip archive project is fetched

Scenario: Zip archive project is fetched
    Given an archive "SomeProject.zip" with the files
        | path          |
        | README.md     |
        | include/lib.h |
    And the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
          version: '0.0'
          projects:
            - name: SomeProject
              url: some-remote-server/SomeProject.zip
              vcs: archive
        """
    When I run "dfetch update" in MyProject
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeProject/
                .dfetch_data.yaml
                README.md
                include/
                    lib.h
            dfetch.yaml
        """

Example: Archive projects with sha256, sha384 and sha512 hash verification are fetched

Scenario: Archive projects with sha256, sha384 and sha512 hash verification are fetched
    Given an archive "SomeProject.tar.gz" with the files
        | path      |
        | README.md |
    And the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
          version: '0.0'
          projects:
            - name: SomeProject-sha256
              url: some-remote-server/SomeProject.tar.gz
              vcs: archive
              integrity:
                hash: sha256:<archive-sha256>
            - name: SomeProject-sha384
              url: some-remote-server/SomeProject.tar.gz
              vcs: archive
              integrity:
                hash: sha384:<archive-sha384>
            - name: SomeProject-sha512
              url: some-remote-server/SomeProject.tar.gz
              vcs: archive
              integrity:
                hash: sha512:<archive-sha512>
        """
    When I run "dfetch update" in MyProject
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeProject-sha256/
                .dfetch_data.yaml
                README.md
            SomeProject-sha384/
                .dfetch_data.yaml
                README.md
            SomeProject-sha512/
                .dfetch_data.yaml
                README.md
            dfetch.yaml
        """

Example: Archive with incorrect sha256 hash is rejected

Scenario: Archive with incorrect sha256 hash is rejected
    Given an archive "SomeProject.tar.gz" with the files
        | path      |
        | README.md |
    And the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
          version: '0.0'
          projects:
            - name: SomeProject
              url: some-remote-server/SomeProject.tar.gz
              vcs: archive
              integrity:
                hash: sha256:0000000000000000000000000000000000000000000000000000000000000000
        """
    When I run "dfetch update" in MyProject
    Then the output shows
        """
        Dfetch (0.14.3)
          SomeProject:
          > Hash mismatch for SomeProject! sha256 expected 0000000000000000000000000000000000000000000000000000000000000000
        """

Example: Specific directory from archive can be fetched

Scenario: Specific directory from archive can be fetched
    Given an archive "SomeProject.tar.gz" with the files
        | path              |
        | src/main.c        |
        | src/lib.c         |
        | tests/test_main.c |
    And the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
          version: '0.0'
          projects:
            - name: SomeProject
              url: some-remote-server/SomeProject.tar.gz
              vcs: archive
              src: src/
        """
    When I run "dfetch update" in MyProject
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeProject/
                .dfetch_data.yaml
                lib.c
                main.c
            dfetch.yaml
        """

Example: Files can be ignored when fetching from archive

Scenario: Files can be ignored when fetching from archive
    Given an archive "SomeProject.tar.gz" with the files
        | path              |
        | README.md         |
        | src/main.c        |
        | tests/test_main.c |
    And the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
          version: '0.0'
          projects:
            - name: SomeProject
              url: some-remote-server/SomeProject.tar.gz
              vcs: archive
              ignore:
                - tests
        """
    When I run "dfetch update" in MyProject
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeProject/
                .dfetch_data.yaml
                README.md
                src/
                    main.c
            dfetch.yaml
        """

Example: Archive is re-fetched when force flag is given

Scenario: Archive is re-fetched when force flag is given
    Given an archive "SomeProject.tar.gz" with the files
        | path      |
        | README.md |
    And the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
          version: '0.0'
          projects:
            - name: SomeProject
              url: some-remote-server/SomeProject.tar.gz
              vcs: archive
        """
    And all projects are updated in MyProject
    When I run "dfetch update --force" in MyProject
    Then the output shows
        """
        Dfetch (0.14.3)
          SomeProject:
          > Fetched some-remote-server/SomeProject.tar.gz
        """

Example: Multiple archive projects are fetched

Scenario: Multiple archive projects are fetched
    Given an archive "LibA.tar.gz" with the files
        | path      |
        | README.md |
    And an archive "LibB.zip" with the files
        | path      |
        | README.md |
    And the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
          version: '0.0'
          projects:
            - name: LibA
              url: some-remote-server/LibA.tar.gz
              vcs: archive

            - name: LibB
              url: some-remote-server/LibB.zip
              vcs: archive
        """
    When I run "dfetch update" in MyProject
    Then the following projects are fetched
        | path           |
        | MyProject/LibA |
        | MyProject/LibB |

Example: Archive with internal relative symlink using .. is fetched safely

Scenario: Archive with internal relative symlink using .. is fetched safely
    Given an archive "SomeProject.tar.gz" with the files
        | path             |
        | README.md        |
        | other/target.mk  |
    And the archive "SomeProject.tar.gz" contains a symlink "sub/dir/link.mk" pointing to "../../other/target.mk"
    And the manifest 'dfetch.yaml' in MyProject
        """
        manifest:
          version: '0.0'
          projects:
            - name: SomeProject
              url: some-remote-server/SomeProject.tar.gz
              vcs: archive
        """
    When I run "dfetch update" in MyProject
    Then 'MyProject' looks like:
        """
        MyProject/
            SomeProject/
                .dfetch_data.yaml
                README.md
                other/
                    target.mk
                sub/
                    dir/
                        link.mk
            dfetch.yaml
        """
    And 'MyProject/SomeProject/sub/dir/link.mk' is a symlink pointing to '../../other/target.mk'

Patch¶

DFetch promotes upstreaming changes, but also allows local changes. These changes can be managed with local patch files. DFetch will apply the patch files in order every time a new upstream version is fetched. The patch file can be specified with the patch: attribute. This can be a single patch file or multiple. Patch files should be UTF-8 encoded files and using forward slashes is encouraged for cross-platform support. The path should be relative to the directory of the manifest.

manifest:
    version: 0.0

    remotes:
    - name: github
      url-base: https://github.com/

    projects:
    - name: cpputest
      vcs: git
      repo-path: cpputest/cpputest
      patch: patches/local_changes.patch

The patch should be generated using the Dfetch Diff command. Alternately the patch can be generated manually as such and should be a relative patch, relative to the fetched projects root.

$ git diff --relative=path/to/project HEAD > my_patch.patch

For more details see the git-diff documentation.

Example: A patch file is generated

Scenario: A patch file is generated
    Given "SomeProject/README.md" in MyProject is changed and committed with
        """
        An important sentence for the README!
        """
    When I run "dfetch diff SomeProject"
    Then the patch file 'MyProject/SomeProject.patch' is generated
        """
        diff --git a/README.md b/README.md
        index 1e65bd6..faa3b21 100644
        --- a/README.md
        +++ b/README.md
        @@ -1 +1,2 @@
        Generated file for SomeProject.git
        +An important sentence for the README!
        """

Example: New files are part of the patch

Scenario: New files are part of the patch
    Given files as '*.tmp' are ignored in git in MyProject
    And "SomeProject/NEWFILE.md" in MyProject is created and committed with
        """
        A completely new tracked file.
        """
    And "SomeProject/NEW_UNCOMMITTED_FILE.md" in MyProject is created
    And "SomeProject/IGNORE_ME.tmp" in MyProject is created
    When I run "dfetch diff SomeProject"
    Then the patch file 'MyProject/SomeProject.patch' is generated
        """
        diff --git a/NEWFILE.md b/NEWFILE.md
        new file mode 100644
        index 0000000..a2d8605
        --- /dev/null
        +++ b/NEWFILE.md
        @@ -0,0 +1 @@
        +A completely new tracked file.

        diff --git a/NEW_UNCOMMITTED_FILE.md b/NEW_UNCOMMITTED_FILE.md
        new file mode 100644
        index 0000000..0ee3895
        --- /dev/null
        +++ NEW_UNCOMMITTED_FILE.md
        @@ -0,0 +1,1 @@
        +Some content

        """

Example: No change is present

Scenario: No change is present
    When I run "dfetch diff SomeProject"
    Then the output shows
    """
    Dfetch (0.14.3)
      SomeProject:
      > No diffs found since 59efb91396fd369eb113b43382783294dc8ed6d2
    """

Example: Diff is generated on uncommitted changes

Scenario: Diff is generated on uncommitted changes
    Given "SomeProject/README.md" in MyProject is changed with
        """
        An important sentence for the README!
        """
    When I run "dfetch diff SomeProject"
    Then the patch file 'MyProject/SomeProject.patch' is generated
        """
        diff --git a/README.md b/README.md
        index 1e65bd6..faa3b21 100644
        --- a/README.md
        +++ b/README.md
        @@ -1 +1,2 @@
        Generated file for SomeProject.git
        +An important sentence for the README!
        """

Example: Metadata is not part of diff

Scenario: Metadata is not part of diff
    Given the metadata file ".dfetch_data.yaml" of "MyProject/SomeProject" is changed
    When I run "dfetch diff SomeProject"
    Then the output shows
    """
    Dfetch (0.14.3)
      SomeProject:
      > No diffs found since 59efb91396fd369eb113b43382783294dc8ed6d2
    """

Example: A patch file is applied after fetching

Scenario: A patch file is applied after fetching
    Given the manifest 'dfetch.yaml'
        """
        manifest:
            version: '0.0'

            remotes:
            - name: github-com-dfetch-org
              url-base: https://github.com/dfetch-org/test-repo

            projects:
            - name: ext/test-repo-tag
              tag: v2.0
              dst: ext/test-repo-tag
              patch: diff.patch
        """
    And the patch file 'diff.patch'
        """
        diff --git a/README.md b/README.md
        index 32d9fad..62248b7 100644
        --- a/README.md
        +++ b/README.md
        @@ -1,2 +1,2 @@
         # Test-repo
        -A test repo for testing dfetch.
        +A test repo for testing patch.
        """
    When I run "dfetch update"
    Then the patched 'ext/test-repo-tag/README.md' is
        """
        # Test-repo
        A test repo for testing patch.
        """

Example: Applying patch file fails

Scenario: Applying patch file fails
    Given the manifest 'dfetch.yaml'
        """
        manifest:
            version: '0.0'

            remotes:
            - name: github-com-dfetch-org
              url-base: https://github.com/dfetch-org/test-repo

            projects:
            - name: ext/test-repo-tag
              tag: v2.0
              dst: ext/test-repo-tag
              patch: diff.patch
        """
    And the patch file 'diff.patch'
        """
        diff --git a/README.md b/README1.md
        index 32d9fad..62248b7 100644
        --- a/README1.md
        +++ b/README1.md
        @@ -1,2 +1,2 @@
         # Test-repo
        -A test repo for testing dfetch.
        +A test repo for testing patch.
        """
    When I run "dfetch update"
    Then the output shows
        """
        Dfetch (0.14.3)
          ext/test-repo-tag:
          > Fetched v2.0
          > Applying patch "diff.patch"
            source/target file does not exist:
              --- b'README1.md'
              +++ b'README1.md'
          > Applying patch "diff.patch" failed
        """

Example: Multiple patch files are applied after fetching

Scenario: Multiple patch files are applied after fetching
    Given the manifest 'dfetch.yaml'
        """
        manifest:
            version: '0.0'

            remotes:
            - name: github-com-dfetch-org
              url-base: https://github.com/dfetch-org/test-repo

            projects:
            - name: ext/test-repo-tag
              tag: v2.0
              dst: ext/test-repo-tag
              patch:
               - 001-diff.patch
               - 002-diff.patch
        """
    And the patch file '001-diff.patch'
        """
        diff --git a/README.md b/README.md
        index 32d9fad..62248b7 100644
        --- a/README.md
        +++ b/README.md
        @@ -1,2 +1,2 @@
         # Test-repo
        -A test repo for testing dfetch.
        +A test repo for testing patch.
        """
    And the patch file '002-diff.patch'
        """
        diff --git a/README.md b/README.md
        index 62248b7..32d9fad 100644
        --- a/README.md
        +++ b/README.md
        @@ -1,2 +1,2 @@
         # Test-repo
        -A test repo for testing patch.
        +A test repo for testing dfetch.
        """
    When I run "dfetch update"
    Then the patched 'ext/test-repo-tag/README.md' is
        """
        # Test-repo
        A test repo for testing dfetch.
        """
    And the output shows
        """
        Dfetch (0.14.3)
          ext/test-repo-tag:
          > Fetched v2.0
          > Applying patch "001-diff.patch"
            successfully patched 1/1:    b'README.md'
          > Applying patch "002-diff.patch"
            successfully patched 1/1:    b'README.md'
        """

Example: Fallback to other file encodings if patch file is not UTF-8 encoded

Scenario: Fallback to other file encodings if patch file is not UTF-8 encoded
    Given the manifest 'dfetch.yaml'
        """
        manifest:
            version: '0.0'

            remotes:
            - name: github-com-dfetch-org
              url-base: https://github.com/dfetch-org/test-repo

            projects:
            - name: ext/test-repo-tag
              tag: v2.0
              dst: ext/test-repo-tag
              patch: diff.patch
        """
    And the patch file 'diff.patch' with 'UTF-16' encoding
        """
        diff --git a/README.md b/README.md
        index 32d9fad..62248b7 100644
        --- a/README.md
        +++ b/README.md
        @@ -1,2 +1,2 @@
         # Test-repo
        -A test repo for testing dfetch.
        +A test repo for testing patch.
        """
    When I run "dfetch update"
    Then the patched 'ext/test-repo-tag/README.md' is
        """
        # Test-repo
        A test repo for testing patch.
        """
    And the output shows
        """
        Dfetch (0.14.3)
          ext/test-repo-tag:
          > Fetched v2.0
          > Applying patch "diff.patch"
            error: no patch data found!
            successfully patched 1/1:    b'README.md'
          > After retrying found that patch-file "diff.patch" is not UTF-8 encoded, consider saving it with UTF-8 encoding.
        """

Example: Patch files are outside manifest dir

Scenario: Patch files are outside manifest dir
    Given the manifest 'dfetch.yaml'
        """
        manifest:
            version: '0.0'

            remotes:
            - name: github-com-dfetch-org
              url-base: https://github.com/dfetch-org/test-repo

            projects:
            - name: ext/test-repo-tag
              tag: v2.0
              dst: ext/test-repo-tag
              patch: ../diff.patch
        """
    When I run "dfetch update"
    Then the output shows
        """
        Dfetch (0.14.3)
          ext/test-repo-tag:
          > Fetched v2.0
          > Skipping patch "../diff.patch" which is outside /some/path.
        """

Example: Patch applies when upstream adds extra lines

Scenario: Patch applies when upstream adds extra lines
    Given "README.md" in git-repository "SomeProject.git" is changed to:
        """
        Generated file for SomeProject.git
        Additional upstream line.
        """
    When I run "dfetch update"
    Then the patched 'MyProject/SomeProject/README.md' is
        """
        Generated file for SomeProject.git
        An important sentence for the README!
        Additional upstream line.
        """

Example: Patch applies when context lines have changed

Scenario: Patch applies when context lines have changed
    Given "README.md" in git-repository "SomeProject.git" is changed to:
        """
        Generated file for SomeProject
        This repository is used for testing.
        """
    And the patch file 'SomeProject.patch' in MyProject
        """
        diff --git a/README.md b/README.md
        index 1e65bd6..faa3b21 100644
        --- a/README.md
        +++ b/README.md
        @@ -1 +1,2 @@
         Generated file for SomeProject.git
        +An important sentence for the README!
        """
    When I run "dfetch update"
    Then the output shows
        """
        Dfetch (0.14.3)
          SomeProject:
          > Fetched master - f47d80c35e14dfa4f9c9c30c9865cbf0f8d50933
          > Applying patch "SomeProject.patch"
            file 1/1:    b'README.md'
             hunk no.1 doesn't match source file at line 1
              expected: b'Generated file for SomeProject.git'
              actual  : b'Generated file for SomeProject'
            successfully patched 1/1:    b'README.md'
        """
    And the patched 'MyProject/SomeProject/README.md' is
        """
        Generated file for SomeProject
        An important sentence for the README!
        This repository is used for testing.
        """

$ svn diff -r HEAD path/to/my_project > my_patch.patch

For more details see the svn-diff documentation.

Example: A patch file is generated

Scenario: A patch file is generated
    Given "SomeProject/README.md" in MySvnProject is changed, added and committed with
        """
        An important sentence for the README!
        """
    When I run "dfetch diff SomeProject" in MySvnProject
    Then the patch file 'MySvnProject/SomeProject.patch' is generated
        """
        Index: README.md
        ===================================================================
        --- README.md
        +++ README.md
        @@ -1,1 +1,2 @@
         Generated file for SomeProject
        +An important sentence for the README!
        """

Example: New files are part of the patch

Scenario: New files are part of the patch
    Given files as '*.tmp' are ignored in 'MySvnProject/SomeProject' in svn
    And "SomeProject/NEWFILE.md" in MySvnProject is changed, added and committed with
        """
        A completely new tracked file.
        """
    And "SomeProject/NEW_UNCOMMITTED_FILE.md" in MySvnProject is created
    And "SomeProject/IGNORE_ME.tmp" in MySvnProject is created
    When I run "dfetch diff SomeProject" in MySvnProject
    Then the patch file 'MySvnProject/SomeProject.patch' is generated
        """
        Index: NEWFILE.md
        ===================================================================
        --- NEWFILE.md
        +++ NEWFILE.md
        @@ -0,0 +1,1 @@
        +A completely new tracked file.
        Index: NEW_UNCOMMITTED_FILE.md
        ===================================================================
        --- /dev/null
        +++ NEW_UNCOMMITTED_FILE.md
        @@ -0,0 +1,1 @@
        +Some content
        """

Example: No change is present

Scenario: No change is present
    When I run "dfetch diff SomeProject" in MySvnProject
    Then the output shows
    """
    Dfetch (0.14.3)
      SomeProject:
      > No diffs found since 1
    """

Example: A patch file is generated on uncommitted changes

Scenario: A patch file is generated on uncommitted changes
    Given "SomeProject/README.md" in MySvnProject is changed with
        """
        An important sentence for the README!
        """
    When I run "dfetch diff SomeProject" in MySvnProject
    Then the patch file 'MySvnProject/SomeProject.patch' is generated
        """
        Index: README.md
        ===================================================================
        --- README.md
        +++ README.md
        @@ -1,1 +1,2 @@
         Generated file for SomeProject
        +An important sentence for the README!
        """

Example: Metadata is not part of diff

Scenario: Metadata is not part of diff
    Given the metadata file ".dfetch_data.yaml" of "MySvnProject/SomeProject" is changed
    When I run "dfetch diff SomeProject" in MySvnProject
    Then the output shows
    """
    Dfetch (0.14.3)
      SomeProject:
      > No diffs found since 1
    """

Example: A patch file is applied after fetching

Scenario: A patch file is applied after fetching
    Given the manifest 'dfetch.yaml'
        """
        manifest:
            version: '0.0'

            remotes:
            - name: cutter
              url-base: https://svn.code.sf.net/p/cutter/svn/cutter

            projects:
            - name: cutter
              vcs: svn
              tag: 1.1.7
              dst: ext/cutter
              patch: diff.patch
              src: apt
        """
    And the patch file 'diff.patch'
        """
        Index: build-deb.sh
        ===================================================================
        --- build-deb.sh	(revision 4007)
        +++ build-deb.sh	(working copy)
        @@ -1,1 +1,1 @@
        -#!/bin/sh
        +#!/bin/bash
        """
    When I run "dfetch update"
    Then the first line of 'ext/cutter/build-deb.sh' is changed to
        """
        #!/bin/bash
        """

Example: Applying patch file fails

Scenario: Applying patch file fails
    Given the manifest 'dfetch.yaml'
        """
        manifest:
            version: '0.0'

            remotes:
            - name: cutter
              url-base: https://svn.code.sf.net/p/cutter/svn/cutter

            projects:
            - name: cutter
              vcs: svn
              tag: 1.1.7
              dst: ext/cutter
              patch: diff.patch
              src: apt
        """
    And the patch file 'diff.patch'
        """
        Index: build-deb.sh
        ===================================================================
        --- build-deb2.sh	(revision 4007)
        +++ build-deb2.sh	(working copy)
        @@ -1,1 +1,1 @@
        -#!/bin/sh
        +#!/bin/bash
        """
    When I run "dfetch update"
    Then the output shows
        """
        Dfetch (0.14.3)
          cutter:
          > Fetched 1.1.7
          > Applying patch "diff.patch"
            source/target file does not exist:
              --- b'build-deb2.sh'
              +++ b'build-deb2.sh'
          > Applying patch "diff.patch" failed
        """

Schema¶

Below an overview of all possible fields on the manifest. The bold items are mandatory.

Note

A string should be a regular string without NULL or control characters.

type	object
properties
manifest	Top-level manifest object. See Manifest for details.
	type	object
	properties
	version	type	string
	remotes	List of remote sources. See Remotes for details. Each remote must be unique by its name.
		type	array
		items	type	object
			properties
			name	A unique name for the remote.
				type	string
			url-base	type	string
			default	type	boolean
	projects	List of projects to Dfetch. See Projects for details. Each project must be unique by its name.
		type	array
		items	type	object
			properties
			name	A unique name for the project.
				type	string
			dst	Destination path to fetch the project to, see Destination for details.
				type	string
			branch	type	string
			tag	type	string
			revision	type	string
			url	type	string
			repo-path	Path within the repository to fetch, see Repo-path for details.
				type	string
			remote	Name of remote to use as base, see Remotes for details.
				type	string
			patch	One or more patches to apply after fetching, see Patch.
				type	string
			vcs	Version control system used by the project. See VCS type for details.
				type	string
				enum	git, svn, archive
			src	Source path within the repository to fetch, see Source for details.
				type	string
			ignore	Files to ignore. See Ignore for details.
				type	array
				items	type	string
			integrity	Integrity verification block for archive dependencies. Only used with `vcs: archive`. Designed for future extension with `sig:` (detached signature URL) and `sig-key:` (signing-key URL or fingerprint) fields alongside `hash:`. See Archive for details.
				type	object
				properties
				hash	Cryptographic hash of the archive file. Format: `<algorithm>:<hex-digest>`. Supported algorithms: `sha256` (64 hex chars), `sha384` (96 hex chars), and `sha512` (128 hex chars). Example: `sha256:e3b0c4…`.
					type	string

Manifest¶

Remotes¶

Projects¶

Revision/Branch/Tag¶

Destination¶

Repo-path¶

Source¶

Ignore¶

VCS type¶

Archive¶

Integrity verification¶

Patch¶

Schema¶

Navigation

Related Topics