Manual¶
Init¶
Initialize a manifest.
usage: dfetch init [-h]
Generate a manifest that can be used as basis for a project.
Dfetch can generate a starting manifest.
It will be created in the current folder.
Check¶
Check all projects for updates.
usage: dfetch check [-h] [--no-recommendations] [--jenkins-json outfile]
[--sarif outfile] [--code-climate outfile]
[<project> ...]
Positional Arguments¶
- <project>
Specific project(s) to check
Named Arguments¶
- --no-recommendations, -N
Ignore recommendations from fetched projects.
Default:
False- --jenkins-json
Generate a JSON that can be parsed by Jenkins.
- --sarif
Generate a Sarif JSON that can be parsed by Github.
- --code-climate
Generate a code-climate JSON that can be parsed by Gitlab.
Check all projects to see if there are any new updates.
To check if your projects are up-to-date, you can let dfetch check it.
For each project the local version (based on tag or revision) will be compared against the available version. If there are new versions available this will be shown.
Example: Git projects have changes
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org/test-repo
projects:
- name: ext/test-repo-rev-only
revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a
dst: ext/test-repo-rev-only
- name: ext/test-rev-and-branch
revision: 8df389d0524863b85f484f15a91c5f2c40aefda1
branch: main
dst: ext/test-rev-and-branch
"""
When I run "dfetch check"
Then the output shows
"""
Dfetch (0.10.0)
ext/test-repo-rev-only: wanted (e1fda19a57b873eb8e6ae37780594cbb77b70f1a), available (e1fda19a57b873eb8e6ae37780594cbb77b70f1a)
ext/test-rev-and-branch: wanted (main - 8df389d0524863b85f484f15a91c5f2c40aefda1), available (main - e1fda19a57b873eb8e6ae37780594cbb77b70f1a)
"""
Example: A newer tag is available than in manifest
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org/test-repo
projects:
- name: ext/test-repo-tag-v1
tag: v1
dst: ext/test-repo-tag-v1
"""
When I run "dfetch check"
Then the output shows
"""
Dfetch (0.10.0)
ext/test-repo-tag-v1: wanted (v1), available (v2.0)
"""
Example: Check is done after an update
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org/test-repo
projects:
- name: ext/test-repo-rev-only
revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a
dst: ext/test-repo-rev-only
- name: ext/test-rev-and-branch
revision: 8df389d0524863b85f484f15a91c5f2c40aefda1
branch: main
dst: ext/test-rev-and-branch
"""
And all projects are updated
When I run "dfetch check"
Then the output shows
"""
Dfetch (0.10.0)
ext/test-repo-rev-only: up-to-date (e1fda19a57b873eb8e6ae37780594cbb77b70f1a)
ext/test-rev-and-branch: wanted & current (main - 8df389d0524863b85f484f15a91c5f2c40aefda1), available (main - e1fda19a57b873eb8e6ae37780594cbb77b70f1a)
"""
Example: Tag is updated in manifest
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
projects:
- name: ext/test-repo-tag
url: https://github.com/dfetch-org/test-repo
tag: v1
"""
And all projects are updated
When the manifest 'dfetch.yaml' is changed to
"""
manifest:
version: '0.0'
projects:
- name: ext/test-repo-tag
url: https://github.com/dfetch-org/test-repo
tag: v2.0
"""
And I run "dfetch check"
Then the output shows
"""
Dfetch (0.10.0)
ext/test-repo-tag : wanted (v2.0), current (v1), available (v2.0)
"""
Example: A local change is reported
Given a git repository "SomeProject.git"
And a fetched and committed MyProject with the manifest
"""
manifest:
version: 0.0
projects:
- name: SomeProject
url: some-remote-server/SomeProject.git
"""
And "SomeProject/README.md" in MyProject is changed and committed with
"""
An important sentence for the README!
"""
When I run "dfetch check SomeProject"
Then the output shows
"""
Dfetch (0.10.0)
SomeProject : Local changes were detected, please generate a patch using 'dfetch diff SomeProject' and add it to your manifest using 'patch:'. Alternatively overwrite the local changes with 'dfetch update --force SomeProject'
SomeProject : up-to-date (master - 90be799b58b10971691715bdc751fbe5237848a0)
"""
Example: A non-existent remote is reported
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
projects:
- name: non-existent-url
revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a
dst: ext/test-repo-rev-only
url: https://giiiiiidhub.com/i-do-not-exist/broken
"""
When I run "dfetch check"
Then the output shows
"""
Dfetch (0.10.0)
>>>git ls-remote --heads https://giiiiiidhub.com/i-do-not-exist/broken<<< failed!
'https://giiiiiidhub.com/i-do-not-exist/broken' is not a valid URL or unreachable:
fatal: unable to access 'https://giiiiiidhub.com/i-do-not-exist/broken/': Could not resolve host: giiiiiidhub.com
"""
Example: A non-existent tag, branch or revision is reported
Given a git repository "SomeProject.git"
And the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
projects:
- name: SomeProjectMissingTag
tag: i-dont-exist
url: some-remote-server/SomeProject.git
- name: SomeProjectNonExistentBranch
branch: i-dont-exist
url: some-remote-server/SomeProject.git
- name: SomeProjectNonExistentRevision
revision: '0123112321234123512361236123712381239123'
url: some-remote-server/SomeProject.git
"""
When I run "dfetch check"
Then the output shows
"""
Dfetch (0.10.0)
SomeProjectMissingTag: wanted (i-dont-exist), but not available at the upstream.
SomeProjectNonExistentBranch: wanted (i-dont-exist), but not available at the upstream.
SomeProjectNonExistentRevision: wanted (0123112321234123512361236123712381239123), but not available at the upstream.
"""
Child-manifests¶
It is possible that fetched projects have manifests of their own.
After these projects are fetched (with dfetch update), the manifests are read as well
and will be checked to look for further dependencies. If you don’t want recommendations,
you can prevent Dfetch from checking child-manifests with --no-recommendations.
Example: Git projects are specified in the manifest
Given the manifest 'dfetch.yaml' in MyProject
"""
manifest:
version: 0.0
projects:
- name: SomeProject
dst: third-party/SomeProject
url: some-remote-server/SomeProject.git
tag: v1
"""
And a git-repository "SomeProject.git" with the manifest:
"""
manifest:
version: 0.0
projects:
- name: SomeOtherProject
dst: ../SomeOtherProject
url: some-remote-server/SomeOtherProject.git
tag: v1
"""
And a git repository "SomeOtherProject.git"
When I run "dfetch check" in MyProject
Then the output shows
"""
Dfetch (0.10.0)
SomeProject : wanted (v1), available (v1)
"""
Example: A recommendation is done due to a missing dependency
Given the manifest 'dfetch.yaml' in MyProject
"""
manifest:
version: 0.0
projects:
- name: SomeProject
dst: third-party/SomeProject
url: some-remote-server/SomeProject.git
tag: v1
"""
And a git-repository "SomeProject.git" with the manifest:
"""
manifest:
version: 0.0
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org/test-repo
projects:
- name: SomeOtherProject
url: some-remote-server/SomeOtherProject.git
tag: v1
- name: ext/test-repo-tag-v1
tag: v1
"""
And a git repository "SomeOtherProject.git"
And all projects are updated in MyProject
When I run "dfetch check" in MyProject
Then the output shows
"""
Dfetch (0.10.0)
Multiple manifests found, using dfetch.yaml
SomeProject : up-to-date (v1)
"SomeProject" depends on the following project(s) which are not part of your manifest:
(found in third-party/SomeProject/dfetch.yaml)
- name: SomeOtherProject
url: some-remote-server/SomeOtherProject.git
tag: v1
- name: ext/test-repo-tag-v1
url: https://github.com/dfetch-org/test-repo
tag: v1
"""
Example: No recommendation is done
Given the manifest 'dfetch.yaml' in MyProject
"""
manifest:
version: 0.0
projects:
- name: SomeProject
dst: third-party/SomeProject
url: some-remote-server/SomeProject.git
tag: v1
- name: SomeOtherProject
url: some-remote-server/SomeOtherProject.git
tag: v1
"""
And a git-repository "SomeProject.git" with the manifest:
"""
manifest:
version: 0.0
projects:
- name: SomeOtherProject
url: some-remote-server/SomeOtherProject.git
"""
And a git repository "SomeOtherProject.git"
And all projects are updated in MyProject
When I run "dfetch check" in MyProject
Then the output shows
"""
Dfetch (0.10.0)
Multiple manifests found, using dfetch.yaml
SomeProject : up-to-date (v1)
SomeOtherProject : up-to-date (v1)
"""
Reporting¶
Report issues found during check.
DFetch can report the results of Check in a format usable by several other tools. See the respective sections for details about using and configuring those reporters.
All reports can contain the following results:
unfetched-projectProject was never fetched. Fetch it using dfetch update.
up-to-date-projectProject is up-to-date.
unavailable-project-versionRequested project version is unavailable at the remote.
pinned-but-out-of-date-projectProject is pinned, but out-of-date. Either ignore this message, or update the version in the manifest.
out-of-date-projectProject is out-of-date. Update the project using dfetch update.
local-changes-in-projectProject was locally changed. Create a patch file using dfetch diff and add it to your manifest using the patch attribute.
Note
When a dfetch check is performed on a different platform than the original
dfetch update the line-endings might result in a false positive of local-changes-in-project.
Jenkins reporter¶
Dfetch can generate a report that is parseable by Jenkins from the Check results.
Depending on the state of the projects, it will create a report with relevant information. If all projects are up-to-date, nothing will be added to the report.
The information has several severities:
high: An unfetched project. Fetch the project to solve the issue.normal: An out-of-date project. The project is not pinned and a newer version is available.lowa pinned but out-of-date project. The project is pinned to a specific version,but a newer version is available.
The report generated is the native json format of the warnings-ng plugin. The plugin will show an overview of the found issues:
When an issue is clicked, you can see the exact location in the manifest where the project is listed.
Example: A newer tag is available than in manifest
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org/test-repo
projects:
- name: ext/test-repo-tag-v1
tag: v1
dst: ext/test-repo-tag-v1
"""
When I run "dfetch check --jenkins-json jenkins.json"
Then the 'jenkins.json' file contains
"""
{
"_class": "io.jenkins.plugins.analysis.core.restapi.ReportApi",
"issues": [
{
"fileName": "dfetch.yaml",
"severity": "High",
"message": "ext/test-repo-tag-v1 : ext/test-repo-tag-v1 was never fetched!",
"description": "The manifest requires version 'v1' of ext/test-repo-tag-v1. it was never fetched, fetch it with 'dfetch update ext/test-repo-tag-v1. The latest version available is 'v2.0'",
"lineStart": 9,
"lineEnd": 9,
"columnStart": 13,
"columnEnd": 32
}
]
}
"""
Example: Check is done after an update
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org/test-repo
projects:
- name: ext/test-repo-rev-only
revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a
dst: ext/test-repo-rev-only
- name: ext/test-rev-and-branch
revision: 8df389d0524863b85f484f15a91c5f2c40aefda1
branch: main
dst: ext/test-rev-and-branch
"""
And all projects are updated
When I run "dfetch check --jenkins-json jenkins.json"
Then the 'jenkins.json' file contains
"""
{
"_class": "io.jenkins.plugins.analysis.core.restapi.ReportApi",
"issues": [
{
"fileName": "dfetch.yaml",
"severity": "Low",
"message": "ext/test-rev-and-branch : ext/test-rev-and-branch wanted & current version is 'main - 8df389d0524863b85f484f15a91c5f2c40aefda1', but 'main - e1fda19a57b873eb8e6ae37780594cbb77b70f1a' is available.",
"description": "The manifest requires version 'main - 8df389d0524863b85f484f15a91c5f2c40aefda1' of ext/test-rev-and-branch. This is also the current version. There is a newer version available 'main - e1fda19a57b873eb8e6ae37780594cbb77b70f1a' You can update the version in the manifest and run 'dfetch update ext/test-rev-and-branch'",
"lineStart": 13,
"lineEnd": 13,
"columnStart": 13,
"columnEnd": 35
}
]
}
"""
Example: Tag is updated in manifest
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
projects:
- name: ext/test-repo-tag
url: https://github.com/dfetch-org/test-repo
tag: v1
"""
And all projects are updated
When the manifest 'dfetch.yaml' is changed to
"""
manifest:
version: '0.0'
projects:
- name: ext/test-repo-tag
url: https://github.com/dfetch-org/test-repo
tag: v2.0
"""
And I run "dfetch check --jenkins-json jenkins.json"
Then the 'jenkins.json' file contains
"""
{
"_class": "io.jenkins.plugins.analysis.core.restapi.ReportApi",
"issues": [
{
"fileName": "dfetch.yaml",
"severity": "Normal",
"message": "ext/test-repo-tag : ext/test-repo-tag current version is 'v1', the wanted version is 'v2.0', but 'v2.0' is available.",
"description": "The manifest requires version 'v2.0' of ext/test-repo-tag. Currently version 'v1' is present. There is a newer version available 'v2.0'. Please update using 'dfetch update ext/test-repo-tag.",
"lineStart": 5,
"lineEnd": 5,
"columnStart": 13,
"columnEnd": 29
}
]
}
"""
Example: A local change is reported
Given a git repository "SomeProject.git"
And a fetched and committed MyProject with the manifest
"""
manifest:
version: 0.0
projects:
- name: SomeProject
url: some-remote-server/SomeProject.git
"""
And "SomeProject/README.md" in MyProject is changed and committed with
"""
An important sentence for the README!
"""
When I run "dfetch check --jenkins-json jenkins.json SomeProject" in MyProject
Then the 'MyProject/jenkins.json' file contains
"""
{
"_class": "io.jenkins.plugins.analysis.core.restapi.ReportApi",
"issues": [
{
"fileName": "dfetch.yaml",
"severity": "Normal",
"message": "SomeProject : SomeProject has local changes, please create a patch file or upstream the changes.",
"description": "SomeProject has local changes, please create a patch file using 'dfetch diff SomeProject. This patch file can either be used to directly from the manifest using the patch attribute, or upstreamed.",
"lineStart": 4,
"lineEnd": 4,
"columnStart": 15,
"columnEnd": 25
}
]
}
"""
Example: Non-existent revision is reported
Given a git repository "SomeProject.git"
And the manifest 'dfetch.yaml'
"""
manifest:
version: 0.0
projects:
- name: SomeProject
url: some-remote-server/SomeProject.git
revision: '0123112321234123512361236123712381239123'
"""
When I run "dfetch check --jenkins-json jenkins.json SomeProject"
Then the 'jenkins.json' file contains
"""
{
"_class": "io.jenkins.plugins.analysis.core.restapi.ReportApi",
"issues": [
{
"fileName": "dfetch.yaml",
"severity": "Low",
"message": "SomeProject : SomeProject wanted version is '0123112321234123512361236123712381239123', but '0123112321234123512361236123712381239123' is unavailable.",
"description": "The manifest requires version '0123112321234123512361236123712381239123' of SomeProject. However the version is unavailable ",
"lineStart": 4,
"lineEnd": 4,
"columnStart": 15,
"columnEnd": 25
}
]
}
"""
Usage¶
Add to pipeline using warnings-ng plugin:
/* For a windows agent */
bat: 'dfetch check --jenkins-json jenkins.json'
/* For a linux agent */
sh: 'dfetch check --jenkins-json jenkins.json'
recordIssues tool: issues(pattern: 'jenkins.json', name: 'DFetch')
With the warnings-ng plugin quality gates thresholds can be set to influence the build result. For example don’t fail when pinned projects are out-of-date.For more information see the quality gate configuration documentation of the warnings-ng plugin.
Sarif reporter¶
Dfetch can generate a report in the Sarif format that is compatible with GitHub from the Check results.
Depending on the state of the projects it will create a report with information. If all project are up-to-date, nothing will be added to the report.
DFetch can be integrated into your GitHub Actions during pull requests.
The found results can be inspected in the run. Below an example of a locally changed project.
By clicking on ‘details’, you can view the project in the manifest.
The information has several severities:
Error: An unfetched project. Fetch the project to solve the issue.Warning: An out-of-date project. The project is not pinned and a newer version is available.Notea pinned but out-of-date project. The project is pinned to a specific version,but a newer version is available.
Example: A newer tag is available than in manifest
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org/test-repo
projects:
- name: ext/test-repo-tag-v1
tag: v1
dst: ext/test-repo-tag-v1
"""
When I run "dfetch check --sarif sarif.json"
Then the 'sarif.json' file contains
"""
{
"runs": [
{
"tool": {
"driver": {
"name": "DFetch",
"informationUri": "https://dfetch.rtfd.io",
"rules": [
{
"id": "unfetched-project",
"help": {
"text": "The project mentioned in the manifest was never fetched, fetch it with 'dfetch update <project>'. After fetching, commit the updated project to your repository."
},
"shortDescription": {
"text": "Project was never fetched"
}
},
{
"id": "up-to-date-project",
"help": {
"text": "The project mentioned in the manifest is up-to-date, everything is ok, nothing to do."
},
"shortDescription": {
"text": "Project is up-to-date"
}
},
{
"id": "unavailable-project-version",
"help": {
"text": "The project mentioned in the manifest is pinned to a specific version, For instance a branch, tag, or revision. However the specific version is not available at the upstream of the project. Check if the remote has the given version. "
},
"shortDescription": {
"text": "Requested project version is unavailable at the remote"
}
},
{
"id": "pinned-but-out-of-date-project",
"help": {
"text": "The project mentioned in the manifest is pinned to a specific version, For instance a branch, tag, or revision. This is currently the state of the project. However a newer version is available at the upstream of the project. Either ignore this warning or update the version to the latest and update using 'dfetch update <project>' and commit the result to your repository."
},
"shortDescription": {
"text": "Project is pinned, but out-of-date"
}
},
{
"id": "out-of-date-project",
"help": {
"text": "The project is configured to always follow the latest version, There is a newer version available at the upstream of the project. Please update the project using 'dfetch update <project>' and commit the result to your repository."
},
"shortDescription": {
"text": "Project is out-of-date"
}
},
{
"id": "local-changes-in-project",
"help": {
"text": "The files of this project are different then when they were added, Please create a patch using 'dfetch diff <project>' and add it to the manifest using the 'patch:' attribute. Or better yet, upstream the changes and update your project. When running 'dfetch check' on a platform with different line endings, then this warning is likely a false positive."
},
"shortDescription": {
"text": "Project was locally changed"
}
}
]
}
},
"artifacts": [
{
"location": {
"uri": "dfetch.yaml"
},
"sourceLanguage": "yaml"
}
],
"results": [
{
"message": {
"text": "ext/test-repo-tag-v1 : ext/test-repo-tag-v1 was never fetched!"
},
"level": "error",
"locations": [
{
"physicalLocation": {
"artifactLocation": {
"index": 0,
"uri": "dfetch.yaml"
},
"region": {
"endColumn": 33,
"endLine": 9,
"startColumn": 13,
"startLine": 9
}
}
}
],
"ruleId": "unfetched-project"
}
]
}
],
"version": "2.1.0"
}
"""
Example: Check is done after an update
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org/test-repo
projects:
- name: ext/test-repo-rev-only
revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a
dst: ext/test-repo-rev-only
- name: ext/test-rev-and-branch
revision: 8df389d0524863b85f484f15a91c5f2c40aefda1
branch: main
dst: ext/test-rev-and-branch
"""
And all projects are updated
When I run "dfetch check --sarif sarif.json"
Then the 'sarif.json' file contains
"""
{
"runs": [
{
"tool": {
"driver": {
"name": "DFetch",
"informationUri": "https://dfetch.rtfd.io",
"rules": [
{
"id": "unfetched-project",
"help": {
"text": "The project mentioned in the manifest was never fetched, fetch it with 'dfetch update <project>'. After fetching, commit the updated project to your repository."
},
"shortDescription": {
"text": "Project was never fetched"
}
},
{
"id": "up-to-date-project",
"help": {
"text": "The project mentioned in the manifest is up-to-date, everything is ok, nothing to do."
},
"shortDescription": {
"text": "Project is up-to-date"
}
},
{
"id": "unavailable-project-version",
"help": {
"text": "The project mentioned in the manifest is pinned to a specific version, For instance a branch, tag, or revision. However the specific version is not available at the upstream of the project. Check if the remote has the given version. "
},
"shortDescription": {
"text": "Requested project version is unavailable at the remote"
}
},
{
"id": "pinned-but-out-of-date-project",
"help": {
"text": "The project mentioned in the manifest is pinned to a specific version, For instance a branch, tag, or revision. This is currently the state of the project. However a newer version is available at the upstream of the project. Either ignore this warning or update the version to the latest and update using 'dfetch update <project>' and commit the result to your repository."
},
"shortDescription": {
"text": "Project is pinned, but out-of-date"
}
},
{
"id": "out-of-date-project",
"help": {
"text": "The project is configured to always follow the latest version, There is a newer version available at the upstream of the project. Please update the project using 'dfetch update <project>' and commit the result to your repository."
},
"shortDescription": {
"text": "Project is out-of-date"
}
},
{
"id": "local-changes-in-project",
"help": {
"text": "The files of this project are different then when they were added, Please create a patch using 'dfetch diff <project>' and add it to the manifest using the 'patch:' attribute. Or better yet, upstream the changes and update your project. When running 'dfetch check' on a platform with different line endings, then this warning is likely a false positive."
},
"shortDescription": {
"text": "Project was locally changed"
}
}
]
}
},
"artifacts": [
{
"location": {
"uri": "dfetch.yaml"
},
"sourceLanguage": "yaml"
}
],
"results": [
{
"message": {
"text": "ext/test-rev-and-branch : ext/test-rev-and-branch wanted & current version is 'main - 8df389d0524863b85f484f15a91c5f2c40aefda1', but 'main - e1fda19a57b873eb8e6ae37780594cbb77b70f1a' is available."
},
"level": "note",
"locations": [
{
"physicalLocation": {
"artifactLocation": {
"index": 0,
"uri": "dfetch.yaml"
},
"region": {
"endColumn": 36,
"endLine": 13,
"startColumn": 13,
"startLine": 13
}
}
}
],
"ruleId": "pinned-but-out-of-date-project"
}
]
}
],
"version": "2.1.0"
}
"""
Example: Tag is updated in manifest
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
projects:
- name: ext/test-repo-tag
url: https://github.com/dfetch-org/test-repo
tag: v1
"""
And all projects are updated
When the manifest 'dfetch.yaml' is changed to
"""
manifest:
version: '0.0'
projects:
- name: ext/test-repo-tag
url: https://github.com/dfetch-org/test-repo
tag: v2.0
"""
And I run "dfetch check --sarif sarif.json"
Then the 'sarif.json' file contains
"""
{
"runs": [
{
"tool": {
"driver": {
"name": "DFetch",
"informationUri": "https://dfetch.rtfd.io",
"rules": [
{
"id": "unfetched-project",
"help": {
"text": "The project mentioned in the manifest was never fetched, fetch it with 'dfetch update <project>'. After fetching, commit the updated project to your repository."
},
"shortDescription": {
"text": "Project was never fetched"
}
},
{
"id": "up-to-date-project",
"help": {
"text": "The project mentioned in the manifest is up-to-date, everything is ok, nothing to do."
},
"shortDescription": {
"text": "Project is up-to-date"
}
},
{
"id": "unavailable-project-version",
"help": {
"text": "The project mentioned in the manifest is pinned to a specific version, For instance a branch, tag, or revision. However the specific version is not available at the upstream of the project. Check if the remote has the given version. "
},
"shortDescription": {
"text": "Requested project version is unavailable at the remote"
}
},
{
"id": "pinned-but-out-of-date-project",
"help": {
"text": "The project mentioned in the manifest is pinned to a specific version, For instance a branch, tag, or revision. This is currently the state of the project. However a newer version is available at the upstream of the project. Either ignore this warning or update the version to the latest and update using 'dfetch update <project>' and commit the result to your repository."
},
"shortDescription": {
"text": "Project is pinned, but out-of-date"
}
},
{
"id": "out-of-date-project",
"help": {
"text": "The project is configured to always follow the latest version, There is a newer version available at the upstream of the project. Please update the project using 'dfetch update <project>' and commit the result to your repository."
},
"shortDescription": {
"text": "Project is out-of-date"
}
},
{
"id": "local-changes-in-project",
"help": {
"text": "The files of this project are different then when they were added, Please create a patch using 'dfetch diff <project>' and add it to the manifest using the 'patch:' attribute. Or better yet, upstream the changes and update your project. When running 'dfetch check' on a platform with different line endings, then this warning is likely a false positive."
},
"shortDescription": {
"text": "Project was locally changed"
}
}
]
}
},
"artifacts": [
{
"location": {
"uri": "dfetch.yaml"
},
"sourceLanguage": "yaml"
}
],
"results": [
{
"message": {
"text": "ext/test-repo-tag : ext/test-repo-tag current version is 'v1', the wanted version is 'v2.0', but 'v2.0' is available."
},
"level": "warning",
"locations": [
{
"physicalLocation": {
"artifactLocation": {
"index": 0,
"uri": "dfetch.yaml"
},
"region": {
"endColumn": 30,
"endLine": 5,
"startColumn": 13,
"startLine": 5
}
}
}
],
"ruleId": "out-of-date-project"
}
]
}
],
"version": "2.1.0"
}
"""
Example: A local change is reported
Given a git repository "SomeProject.git"
And a fetched and committed MyProject with the manifest
"""
manifest:
version: 0.0
projects:
- name: SomeProject
url: some-remote-server/SomeProject.git
"""
And "SomeProject/README.md" in MyProject is changed and committed with
"""
An important sentence for the README!
"""
When I run "dfetch check --sarif sarif.json SomeProject" in MyProject
Then the 'MyProject/sarif.json' file contains
"""
{
"runs": [
{
"tool": {
"driver": {
"name": "DFetch",
"informationUri": "https://dfetch.rtfd.io",
"rules": [
{
"id": "unfetched-project",
"help": {
"text": "The project mentioned in the manifest was never fetched, fetch it with 'dfetch update <project>'. After fetching, commit the updated project to your repository."
},
"shortDescription": {
"text": "Project was never fetched"
}
},
{
"id": "up-to-date-project",
"help": {
"text": "The project mentioned in the manifest is up-to-date, everything is ok, nothing to do."
},
"shortDescription": {
"text": "Project is up-to-date"
}
},
{
"id": "unavailable-project-version",
"help": {
"text": "The project mentioned in the manifest is pinned to a specific version, For instance a branch, tag, or revision. However the specific version is not available at the upstream of the project. Check if the remote has the given version. "
},
"shortDescription": {
"text": "Requested project version is unavailable at the remote"
}
},
{
"id": "pinned-but-out-of-date-project",
"help": {
"text": "The project mentioned in the manifest is pinned to a specific version, For instance a branch, tag, or revision. This is currently the state of the project. However a newer version is available at the upstream of the project. Either ignore this warning or update the version to the latest and update using 'dfetch update <project>' and commit the result to your repository."
},
"shortDescription": {
"text": "Project is pinned, but out-of-date"
}
},
{
"id": "out-of-date-project",
"help": {
"text": "The project is configured to always follow the latest version, There is a newer version available at the upstream of the project. Please update the project using 'dfetch update <project>' and commit the result to your repository."
},
"shortDescription": {
"text": "Project is out-of-date"
}
},
{
"id": "local-changes-in-project",
"help": {
"text": "The files of this project are different then when they were added, Please create a patch using 'dfetch diff <project>' and add it to the manifest using the 'patch:' attribute. Or better yet, upstream the changes and update your project. When running 'dfetch check' on a platform with different line endings, then this warning is likely a false positive."
},
"shortDescription": {
"text": "Project was locally changed"
}
}
]
}
},
"artifacts": [
{
"location": {
"uri": "dfetch.yaml"
},
"sourceLanguage": "yaml"
}
],
"results": [
{
"message": {
"text": "SomeProject : SomeProject has local changes, please create a patch file or upstream the changes."
},
"level": "warning",
"locations": [
{
"physicalLocation": {
"artifactLocation": {
"index": 0,
"uri": "dfetch.yaml"
},
"region": {
"endColumn": 26,
"endLine": 4,
"startColumn": 15,
"startLine": 4
}
}
}
],
"ruleId": "local-changes-in-project"
}
]
}
],
"version": "2.1.0"
}
"""
Usage¶
A Sarif report can be added to a github action as such:
name: DFetch
on: push
jobs:
dfetch:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Set up python 3.10
uses: actions/setup-python@v2
with:
python-version: "3.10"
- name: Set up dfetch
run: |
python -m pip install --upgrade pip
pip install dfetch
- name: Check dependencies
run: dfetch check --sarif sarif.json
- name: Upload SARIF file
uses: github/codeql-action/upload-sarif@v1
with:
sarif_file: sarif.json
For more information see the Github Sarif documentation.
Code-climate reporter¶
Dfetch can generate a report that is parsable by GitLab/Code Climate from the Check results.
Depending on the state of the projects it will create a report with information. If all project are up-to-date, nothing will be added to the report.
The information has several severities:
major: An unfetched project. Fetch the project to solve the issue.minor: An out-of-date project. The project is not pinned and a newer version is available.infoa pinned but out-of-date project. The project is pinned to a specific version,but a newer version is available.
The report generated is the code-climate json format, the fields described in the gitlab custom code quality tool documentation are in the report but also all fields listed as required by the code-climate json format.
Gitlab will show the results with the pipeline
And clicking will bring you to the project in the manifest.
In the merge request, GitLab will compare the issues in the current branch with those of
the base branch (e.g. master/main). This lets you see if any new issue were introduced
or solved.
Example: A newer tag is available than in manifest
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org/test-repo
projects:
- name: ext/test-repo-tag-v1
tag: v1
dst: ext/test-repo-tag-v1
"""
When I run "dfetch check --code-climate code_climate.json"
Then the 'code_climate.json' file contains
"""
[
{
"description": "The manifest requires version 'v1' of ext/test-repo-tag-v1. it was never fetched, fetch it with 'dfetch update ext/test-repo-tag-v1. The latest version available is 'v2.0'",
"check_name": "unfetched-project",
"categories": [
"Security",
"Bug risk"
],
"fingerprint": "2cc525a6d825f2dc541ade86dfdd351a1a7c47c2b41f3d31fa9b7b19f9400006",
"severity": "major",
"location": {
"path": "dfetch.yaml",
"positions": {
"begin": {
"line": 9,
"column": 13
},
"end": {
"line": 9,
"column": 32
}
}
}
}
]
"""
Example: Check is done after an update
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org/test-repo
projects:
- name: ext/test-repo-rev-only
revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a
dst: ext/test-repo-rev-only
- name: ext/test-rev-and-branch
revision: 8df389d0524863b85f484f15a91c5f2c40aefda1
branch: main
dst: ext/test-rev-and-branch
"""
And all projects are updated
When I run "dfetch check --code-climate code_climate.json"
Then the 'code_climate.json' file contains
"""
[
{
"description": "The manifest requires version 'main - 8df389d0524863b85f484f15a91c5f2c40aefda1' of ext/test-rev-and-branch. This is also the current version. There is a newer version available 'main - e1fda19a57b873eb8e6ae37780594cbb77b70f1a' You can update the version in the manifest and run 'dfetch update ext/test-rev-and-branch'",
"check_name": "pinned-but-out-of-date-project",
"categories": [
"Security",
"Bug risk"
],
"fingerprint": "ea830628ce63a7d3331e72ce95fa7cd7b2017ed2f17c1e77445d7a3cde140af3",
"severity": "info",
"location": {
"path": "dfetch.yaml",
"positions": {
"begin": {
"line": 13,
"column": 13
},
"end": {
"line": 13,
"column": 35
}
}
}
}
]
"""
Example: Tag is updated in manifest
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
projects:
- name: ext/test-repo-tag
url: https://github.com/dfetch-org/test-repo
tag: v1
"""
And all projects are updated
When the manifest 'dfetch.yaml' is changed to
"""
manifest:
version: '0.0'
projects:
- name: ext/test-repo-tag
url: https://github.com/dfetch-org/test-repo
tag: v2.0
"""
And I run "dfetch check --code-climate code_climate.json"
Then the 'code_climate.json' file contains
"""
[
{
"description": "The manifest requires version 'v2.0' of ext/test-repo-tag. Currently version 'v1' is present. There is a newer version available 'v2.0'. Please update using 'dfetch update ext/test-repo-tag.",
"check_name": "out-of-date-project",
"categories": [
"Security",
"Bug risk"
],
"fingerprint": "14879c573e42bdc3ef8feb7ea73f23b0894839b859b713dabaf4efa1b4f2537a",
"severity": "minor",
"location": {
"path": "dfetch.yaml",
"positions": {
"begin": {
"line": 5,
"column": 13
},
"end": {
"line": 5,
"column": 29
}
}
}
}
]
"""
Example: A local change is reported
Given a git repository "SomeProject.git"
And a fetched and committed MyProject with the manifest
"""
manifest:
version: 0.0
projects:
- name: SomeProject
url: some-remote-server/SomeProject.git
"""
And "SomeProject/README.md" in MyProject is changed and committed with
"""
An important sentence for the README!
"""
When I run "dfetch check --code-climate code_climate.json SomeProject" in MyProject
Then the 'MyProject/code_climate.json' file contains
"""
[
{
"description": "SomeProject has local changes, please create a patch file using 'dfetch diff SomeProject. This patch file can either be used to directly from the manifest using the patch attribute, or upstreamed.",
"check_name": "local-changes-in-project",
"categories": [
"Security",
"Bug risk"
],
"fingerprint": "c9e65f37627d383daf06893c932080d554ef8b9578b81e6d2914f04f0a18c120",
"severity": "minor",
"location": {
"path": "dfetch.yaml",
"positions": {
"begin": {
"line": 4,
"column": 15
},
"end": {
"line": 4,
"column": 25
}
}
}
}
]
"""
Usage¶
Let DFetch perform a check and generate the code-climate json and add the result as artifact in you gitlab-ci runner. See gitlab code quality reports for more information.
dfetch:
image: "python:3.7"
script:
- pip install dfetch
- dfetch check --code-climate dfetch.json
artifacts:
reports:
codequality: dfetch.json
Report¶
Generate reports containing information about the projects components.
usage: dfetch report [-h] [-o <filename>] [-t {sbom,list}] [<project> ...]
Positional Arguments¶
- <project>
Specific project(s) to report
Named Arguments¶
- -o, --outfile
Report filename
Default:
'report.json'- -t, --type
Possible choices: sbom, list
Type of report to generate.
Default:
list
Report can be stdout, sbom
Dfetch can generate multiple reports.
There are several report types that DFetch can generate.
List (default)¶
Dfetch can generate an report on stdout.
Depending on the state of the projects it will show as much information
from the manifest or the metadata (.dfetch_data.yaml).
Software Bill-of-Materials¶
Dfetch can generate a software Bill-of-Materials (SBOM).
An SBOM lists the components and their supply chain relationships. Downstream users of the software can assess the licenses used and potential risk of dependencies.
The generated SBOM can be used as input for other tools to monitor dependencies. The tools track vulnerabilities or can enforce a license policy within an organization.
See https://cyclonedx.org/use-cases/ for more details.
Dfetch will try to parse the license of project, this is retained during an Update.
Example: An fetched project generates an sbom
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
projects:
- name: cpputest
url: https://github.com/cpputest/cpputest
tag: v3.4
src: 'include/CppUTest'
"""
And all projects are updated
When I run "dfetch report -t sbom"
Then the 'report.json' file contains
"""
{
"$schema": "http://cyclonedx.org/schema/bom-1.4.schema.json",
"bomFormat": "CycloneDX",
"specVersion": "1.4",
"serialNumber": "urn:uuid:e989dc42-a199-4fe4-87f1-2b7f7a5f48cf",
"version": 1,
"dependencies": [
{
"ref": "a3aff0d8-2f40-4482-bded-577466c0bde9"
}
],
"metadata": {
"timestamp": "2023-03-25T19:15:03.697694+00:00",
"tools": [
{
"vendor": "dfetch-org",
"name": "dfetch",
"version": "0.10.0"
},
{
"vendor": "CycloneDX",
"name": "cyclonedx-python-lib",
"version": "4.2.2",
"externalReferences": [
{
"url": "https://pypi.org/project/cyclonedx-python-lib/",
"type": "distribution"
},
{
"url": "https://cyclonedx.org",
"type": "website"
},
{
"url": "https://github.com/CycloneDX/cyclonedx-python-lib/actions",
"type": "build-system"
},
{
"url": "https://github.com/CycloneDX/cyclonedx-python-lib/blob/main/LICENSE",
"type": "license"
},
{
"url": "https://github.com/CycloneDX/cyclonedx-python-lib/blob/main/CHANGELOG.md",
"type": "release-notes"
},
{
"url": "https://cyclonedx.github.io/cyclonedx-python-lib/",
"type": "documentation"
},
{
"url": "https://github.com/CycloneDX/cyclonedx-python-lib/issues",
"type": "issue-tracker"
},
{
"url": "https://github.com/CycloneDX/cyclonedx-python-lib",
"type": "vcs"
}
]
}
]
},
"components": [
{
"type": "library",
"bom-ref": "a3aff0d8-2f40-4482-bded-577466c0bde9",
"name": "cpputest",
"version": "v3.4",
"licenses": [
{
"expression": "BSD 3-Clause \"New\" or \"Revised\" License"
}
],
"purl": "pkg:github/cpputest/cpputest@v3.4#include/CppUTest"
}
]
}
"""
Update¶
Update all modules from the manifest.
usage: dfetch update [-h] [-f] [-N] [<project> ...]
Positional Arguments¶
- <project>
Specific project(s) to update
Named Arguments¶
- -f, --force
Always perform update, ignoring version check or local changes.
Default:
False- -N, --no-recommendations
Ignore recommendations from fetched projects.
Default:
False
Verifies the manifest and checks all dependencies if updates are available.
Update is the main functionality of dfetch.
You can add Projects to your Manifest and update will fetch the version specified. It tries to determine what kind of vcs it is: git, svn or something else.
Example: Git projects are specified in the manifest
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org/test-repo
projects:
- name: ext/test-repo-rev-only
revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a
dst: ext/test-repo-rev-only
- name: ext/test-rev-and-branch
revision: 8df389d0524863b85f484f15a91c5f2c40aefda1
branch: main
dst: ext/test-rev-and-branch
- name: ext/test-repo-tag-v1
tag: v1
dst: ext/test-repo-tag-v1
"""
When I run "dfetch update"
Then the following projects are fetched
| path |
| ext/test-repo-rev-only |
| ext/test-rev-and-branch |
| ext/test-repo-tag-v1 |
Example: Tag is updated in manifest
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
projects:
- name: ext/test-repo-tag
url: https://github.com/dfetch-org/test-repo
tag: v1
"""
And all projects are updated
When the manifest 'dfetch.yaml' is changed to
"""
manifest:
version: '0.0'
projects:
- name: ext/test-repo-tag
url: https://github.com/dfetch-org/test-repo
tag: v2.0
"""
And I run "dfetch update"
Then the output shows
"""
Dfetch (0.10.0)
ext/test-repo-tag : Fetched v2.0
"""
Example: Version check ignored when force flag is given
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
projects:
- name: ext/test-repo-tag
url: https://github.com/dfetch-org/test-repo
tag: v1
"""
And all projects are updated
When I run "dfetch update --force"
Then the output shows
"""
Dfetch (0.10.0)
ext/test-repo-tag : Fetched v1
"""
Child-manifests¶
It is possible that fetched projects have manifests of their own.
When these projects are fetched (with dfetch update), the manifests are read as well
and will be checked to look for further dependencies. If you don’t what recommendations, you can prevent Dfetch
checking child-manifests with --no-recommendations.
Example: Git projects are specified in the manifest
Given the manifest 'dfetch.yaml' in MyProject
"""
manifest:
version: 0.0
projects:
- name: SomeProjectWithChild
dst: third-party/SomeProjectWithChild
url: some-remote-server/SomeProjectWithChild.git
tag: v1
- name: SomeProjectWithoutChild
dst: third-party/SomeProjectWithoutChild
url: some-remote-server/SomeProjectWithoutChild.git
tag: v1
"""
And a git-repository "SomeProjectWithChild.git" with the manifest:
"""
manifest:
version: 0.0
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org/test-repo
projects:
- name: SomeOtherProject
dst: SomeOtherProject
url: some-remote-server/SomeOtherProject.git
tag: v1
- name: ext/test-repo-tag-v1
tag: v1
"""
And a git repository "SomeProjectWithoutChild.git"
When I run "dfetch update" in MyProject
Then the output shows
"""
Dfetch (0.10.0)
SomeProjectWithChild: Fetched v1
"SomeProjectWithChild" depends on the following project(s) which are not part of your manifest:
(found in third-party/SomeProjectWithChild/dfetch.yaml)
- name: SomeOtherProject
url: some-remote-server/SomeOtherProject.git
tag: v1
- name: ext/test-repo-tag-v1
url: https://github.com/dfetch-org/test-repo
tag: v1
SomeProjectWithoutChild: Fetched v1
"""
And 'MyProject' looks like:
"""
MyProject/
dfetch.yaml
third-party/
SomeProjectWithChild/
.dfetch_data.yaml
README.md
dfetch.yaml
SomeProjectWithoutChild/
.dfetch_data.yaml
README.md
"""
Example: A child-project has an invalid manifest
Given the manifest 'dfetch.yaml' in MyProject
"""
manifest:
version: 0.0
projects:
- name: SomeProject
dst: third-party/SomeProject
url: some-remote-server/SomeProject.git
tag: v1
"""
And a git-repository "SomeProject.git" with the manifest:
"""
very-invalid-manifest
"""
When I run "dfetch update" in MyProject
Then the output shows
"""
Dfetch (0.10.0)
SomeProject : Fetched v1
SomeProject/dfetch.yaml: Schema validation failed:
- Value 'very-invalid-manifest' is not a dict. Value path: ''.
"""
And 'MyProject' looks like:
"""
MyProject/
dfetch.yaml
third-party/
SomeProject/
.dfetch_data.yaml
README.md
dfetch.yaml
"""
Validate¶
Validate a manifest.
usage: dfetch validate [-h]
The Manifest is validated against a schema. See manifest for requirements. Note that each time either
updateorcheckis run, the manifest is also validated.
Note that you can validate your manifest using Validate.
This will parse your Manifest and check if all fields can be parsed.
Example: A valid manifest is provided
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org/test-repo
projects:
- name: ext/test-repo-rev-only
revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a
"""
When I run "dfetch validate"
Then the output shows
"""
Dfetch (0.10.0)
dfetch.yaml : valid
"""
Example: An invalid manifest is provided
Given the manifest 'dfetch.yaml'
"""
manifest-wrong:
version: '0.0'
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org/test-repo
projects:
- name: ext/test-repo-rev-only
revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a
"""
When I run "dfetch validate"
Then the output shows
"""
Dfetch (0.10.0)
Schema validation failed:
- Cannot find required key 'manifest'. Path: ''.
- Key 'manifest-wrong' was not defined. Path: ''.
"""
Diff¶
Generate a diff of a project.
usage: dfetch diff [-h] [-r <oldrev>[:<newrev>]] <project>
Positional Arguments¶
- <project>
Project to generate diff from
Named Arguments¶
- -r, --revs
Revision(s) to generate diff from
Default:
''
Create a patch of a project. The diff will be a relative patch file of only the project’s directory.
Dfetch can create a patch file with your local changes to the external project.
If you’ve found any issues with the remote project, you can fix them within the context of your project. To help the upstream project, you can generate a patch file that can be applied by the upstream maintainer. The patch will be generated using the version control system of your main project that contains the manifest.
To generate a patch, Dfetch requires two revisions to identify the changes. You can
provide these through the --revs argument.
If
--revsis not provided the changes are calculated between the last revision the metadata file was changed and the working copy (also uncommitted changes).If
--revsspecifies one revision (e.g.--revs 23864ef2), that revision will be used as starting point.Alternately both revisions can be explicitly specified, e.g.
--revs 23864ef2:4a9cb18.
The below statement will generate a patch for some-project from your manifest.
dfetch diff some-project
Example: A patch file is generated
Given "SomeProject/README.md" in MyProject is changed and committed with
"""
An important sentence for the README!
"""
When I run "dfetch diff SomeProject"
Then the patch file 'MyProject/SomeProject.patch' is generated
"""
diff --git a/README.md b/README.md
index 1e65bd6..faa3b21 100644
--- a/README.md
+++ b/README.md
@@ -1 +1,2 @@
Generated file for SomeProject.git
+An important sentence for the README!
"""
Example: No change is present
When I run "dfetch diff SomeProject"
Then the output shows
"""
Dfetch (0.10.0)
SomeProject : No diffs found since 59efb91396fd369eb113b43382783294dc8ed6d2
"""
Example: Diff is generated on uncommitted changes
Given "SomeProject/README.md" in MyProject is changed with
"""
An important sentence for the README!
"""
When I run "dfetch diff SomeProject"
Then the patch file 'MyProject/SomeProject.patch' is generated
"""
diff --git a/README.md b/README.md
index 1e65bd6..faa3b21 100644
--- a/README.md
+++ b/README.md
@@ -1 +1,2 @@
Generated file for SomeProject.git
+An important sentence for the README!
"""
Using the generated patch¶
The patch can be used in the manifest; see the Patch attribute for more information. It can also be sent to the upstream maintainer in case of bug fixes.
The generated patch is a relative patch and should be by applied specifying the base directory of the git repo. See below for the version control specifics. The patch will also contain the content of binary files.
# For git repo's
git apply --verbose --directory='some-project' some-project.patch
# For svn repo's
svn patch some-project.patch
Warning
The path given to --directory when applying the patch in a git repo, must be relative to the base
directory of the repo, i.e. the folder where the .git folder is located.
For example if you have the patch Core/MyModule/MySubmodule.patch
for files in the directory Core/MyModule/MySubmodule/ and your current working directory is Core/MyModule/.
The correct command would be:
git apply --verbose --directory='Core/MyModule/MySubModule` MySubmodule.patch
Freeze¶
Freeze your projects versions in the manifest as they are on disk.
usage: dfetch freeze [-h]
Generate a new manifest that has all version as they are on disk.
Dfetch can freeze the current versions of the projects.
Say you have the following manifest:
manifest:
version: 0.0
projects:
- name: mymodule
url: http://git.mycompany.local/mycompany/mymodule
As explained in Revision/Branch/Tag when no version is provided the latest
version of the default branch (e.g. trunk, master) of mymodule will
be fetched on a DFetch update.
When your project becomes stable and you want to rely on a specific version
of mymodule you can run dfetch freeze.
First DFetch will rename your old manifest (appended with .backup).
After that a new manifest is generated with all the projects as in your original
manifest, but each with the specific version as it currently is on disk.
In our above example this would for instance result in:
manifest:
version: 0.0
projects:
- name: mymodule
url: http://git.mycompany.local/mycompany/mymodule
tag: v1.0.0
Example: Git projects are frozen
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
projects:
- name: ext/test-repo-tag
url: https://github.com/dfetch-org/test-repo
branch: main
"""
And all projects are updated
When I run "dfetch freeze"
Then the manifest 'dfetch.yaml' is replaced with
"""
manifest:
version: '0.0'
projects:
- name: ext/test-repo-tag
revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a
url: https://github.com/dfetch-org/test-repo
branch: main
"""
@remote-svn
Example: SVN projects are specified in the manifest
Given the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
projects:
- name: cunit-svn
vcs: svn
url: svn://svn.code.sf.net/p/cunit/code
"""
And all projects are updated
When I run "dfetch freeze"
Then the manifest 'dfetch.yaml' is replaced with
"""
manifest:
version: '0.0'
projects:
- name: cunit-svn
revision: '170'
url: svn://svn.code.sf.net/p/cunit/code
branch: trunk
vcs: svn
"""
Environment¶
Get information about the environment dfetch is working in.
usage: dfetch environment [-h]
Generate output that can be used by dfetch developers to investigate issues.
Dfetch can generate output about its working environment.
Import¶
Generate manifest from existing submodules or externals.
usage: dfetch import [-h]
Look for submodules in a git project or externals in a svn project and create a manifest based on that.
Dfetch can convert your Git submodules or SVN externals based project to Dfetch.
Dfetch will look for all submodules or externals in the current project and generate a manifest with the current versions of the repository.
After importing, you will need to remove the submodules or externals, and then you can let Dfetch update by running dfetch update.
Migrating from git submodules¶
Make sure your repository is up-to-date.
Make sure your submodules are up-to-date (
git submodules update --init).Generate a manifest using dfetch import.
Remove all git submodules (see How do I remove a submodule ).
Download all your projects using dfetch update.
Commit your projects as part of your project.
Example: Multiple submodules are imported
Given a git repo with the following submodules
| path | url | revision |
| ext/test-repo1 | https://github.com/dfetch-org/test-repo | e1fda19a57b873eb8e6ae37780594cbb77b70f1a |
| ext/test-repo2 | https://github.com/dfetch-org/test-repo | 8df389d0524863b85f484f15a91c5f2c40aefda1 |
When I run "dfetch import"
Then it should generate the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org
projects:
- name: ext/test-repo1
revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a
branch: main
repo-path: test-repo
- name: ext/test-repo2
revision: 8df389d0524863b85f484f15a91c5f2c40aefda1
tag: v1
repo-path: test-repo
"""
Switching branches¶
After importing submodules into a manifest within a branch, you might encounter difficulties when switching branches.
If one branch has submodules located where your DFetched project dependencies should be, or vice versa.
In both situations below, assume a branch feature/use-dfetch with a manifest and master with the original
submodules in their place.
Switching from branch with submodules to branch with manifest¶
When switching from branch with submodules to one without, git will warn that the Dfetched project dependencies will overwrite the submodule that is currently in the same location.
$ git checkout feature/use-dfetch
error: The following untracked working tree files would be overwritten by checkout:
MySubmodule/somefile.c
MySubmodule/someotherfile.c
However, git status will show nothing:
$ git status
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
To overcome this, remove the submodule folder and checkout branch feature/use-dfetch:
$ rm -rf MySubmodule
$ git checkout feature/use-dfetch
Switching from branch with manifest to branch with submodules¶
This situation gives no problem, but the submodules are gone and need to be initialized again. To solve this:
$ git checkout master
$ git submodule update --init
Submodule path 'MySubmodule': checked out '08f95e01b297d8b8c1c9101bde58e75cd4d428ce'
Migrating from SVN externals¶
Make sure your repository is up-to-date.
Generate a manifest using dfetch import.
Remove all svn externals (see How do I remove svn::externals ).
Download all your projects using dfetch update.
Commit your projects as part of your project.
Example: Multiple externals are imported
Given a svn repo with the following externals
| path | url | revision |
| ext/test-repo1 | https://github.com/dfetch-org/test-repo/trunk | 1 |
| ext/test-repo2 | https://github.com/dfetch-org/test-repo/tags/v2.0 | |
| ext/test-repo3 | https://github.com/dfetch-org/test-repo | |
When I run "dfetch import"
Then it should generate the manifest 'dfetch.yaml'
"""
manifest:
version: '0.0'
remotes:
- name: github-com-dfetch-org
url-base: https://github.com/dfetch-org
projects:
- name: ext/test-repo1
revision: '1'
dst: ./ext/test-repo1
repo-path: test-repo
- name: ext/test-repo2
dst: ./ext/test-repo2
tag: v2.0
repo-path: test-repo
- name: ext/test-repo3
dst: ./ext/test-repo3
branch: ' '
repo-path: test-repo
"""
