Migrate to Dfetch

Dfetch can convert an existing project that uses Git submodules or SVN externals into a Dfetch-managed project. The dfetch import command reads your current dependency configuration and writes a Manifest — after that you remove the old mechanism and let dfetch update take over.

Choose the guide that matches your current setup:

• From Git submodules

• From SVN externals

From Git submodules

Note

Before you start, make sure your repository is fully up-to-date and all submodules are initialised and checked out:

$ git pull
$ git submodule update --init --recursive

Steps

1. Generate a manifest from the existing submodules:

   $ dfetch import
   

   This writes a dfetch.yaml file in the current directory listing each submodule as a Dfetch project entry, pinned to the commit that is currently checked out.

2. Remove all Git submodules. For each submodule (replace <path> with the submodule path, e.g. ext/mylib):

   $ git submodule deinit -f <path>
   $ git rm -f <path>
   $ rm -rf .git/modules/<path>
   

   Repeat until git submodule status returns nothing. Commit the result:

   $ git commit -m "chore: remove git submodules (switching to Dfetch)"
   

   See also

   How do I remove a submodule?

3. Fetch all projects into your repository:

   $ dfetch update
   

4. Commit the fetched files:

   $ git add .
   $ git commit -m "chore: vendor dependencies with Dfetch"
   

Example: Multiple submodules are imported

../features/import-from-git.feature

Given a git repo with the following submodules
    | path           | url                                     | revision                                 |
    | ext/test-repo1 | https://github.com/dfetch-org/test-repo | e1fda19a57b873eb8e6ae37780594cbb77b70f1a |
    | ext/test-repo2 | https://github.com/dfetch-org/test-repo | 8df389d0524863b85f484f15a91c5f2c40aefda1 |
When I run "dfetch import"
Then it should generate the manifest 'dfetch.yaml'
    """
    manifest:
      version: '0.0'

      remotes:
      - name: github-com-dfetch-org
        url-base: https://github.com/dfetch-org

      projects:
      - name: ext/test-repo1
        revision: e1fda19a57b873eb8e6ae37780594cbb77b70f1a
        branch: main
        repo-path: test-repo

      - name: ext/test-repo2
        revision: 8df389d0524863b85f484f15a91c5f2c40aefda1
        tag: v1
        repo-path: test-repo

    """

Switching branches after migration

If you migrate on a feature branch while master still has the original submodules, you will run into a few rough edges when switching back and forth.

From submodule branch → manifest branch

Git will refuse to check out if a Dfetched dependency would overwrite a submodule directory that is still in place:

$ git checkout feature/use-dfetch
error: The following untracked working tree files would be overwritten by checkout:
    MySubmodule/somefile.c
    MySubmodule/someotherfile.c

git status shows a clean tree — the files belong to the submodule, which Git does not track as ordinary working-tree files.

To work around this, delete the directory before switching:

$ rm -rf MySubmodule
$ git checkout feature/use-dfetch

If you have several submodules, remove them all before checking out.

From manifest branch → submodule branch

This direction succeeds without errors, but Git leaves the submodule directories empty. After checking out, re-initialise them:

$ git checkout master
$ git submodule update --init --recursive
Submodule path 'MySubmodule': checked out '08f95e01b297d8b8c1c9101bde58e75cd4d428ce'

From SVN externals

Before you start, make sure your working copy is fully up-to-date:

$ svn update

Steps

1. Generate a manifest from the existing externals:

   $ dfetch import
   

   This writes a dfetch.yaml listing every svn:externals entry as a Dfetch project, pinned to the revision that is currently set.

2. Remove all SVN externals. Externals are stored as a property on a directory — you need to delete that property for every directory that has one.

   List all directories with svn:externals set:

   $ svn proplist -R | grep -B1 svn:externals
   

   For each such directory (replace <directory>):

   $ svn propdel svn:externals <directory>
   

   Commit the property removal:

   $ svn commit -m "Remove SVN externals (switching to Dfetch)"
   

   See also

   How do I remove SVN externals?

   Note

   If your externals contain nested externals, dfetch import only reads the top-level svn:externals property. Run dfetch import inside each nested external directory and merge the resulting entries into your top-level dfetch.yaml by hand.

3. Fetch all projects into your working copy:

   $ dfetch update
   

4. Commit the fetched files:

   $ svn add --force .
   $ svn commit -m "Vendor dependencies with Dfetch"
   

Example: Multiple externals are imported

../features/import-from-svn.feature

Given a svn repo with the following externals
    | path       | url                                                       | revision       |
    | ext/cunit1 | https://svn.code.sf.net/p/cunit/code/trunk/Man            | 176            |
    | ext/cunit2 | https://svn.code.sf.net/p/cunit/code/branches/mingw64/Man | 150            |
    | ext/cunit3 | https://svn.code.sf.net/p/cunit/code                      |                |
When I run "dfetch import"
Then it should generate the manifest 'dfetch.yaml'
    """
    manifest:
      version: '0.0'

      remotes:
      - name: svn-code-sf-net-p-cunit
        url-base: https://svn.code.sf.net/p/cunit

      projects:
      - name: ext/cunit1
        revision: '176'
        src: Man
        dst: ./ext/cunit1
        repo-path: code

      - name: ext/cunit2
        revision: '150'
        src: Man
        dst: ./ext/cunit2
        branch: mingw64
        repo-path: code

      - name: ext/cunit3
        dst: ./ext/cunit3
        branch: ' '
        repo-path: code

    """