Problem: migration from boltdb to postgresql crashes on user creation - semaphore-2.18.2

[helioloureiro]

Issue

When I attempt to migrate to postgres, it gets and error:

# ./semaphore-2.18.2 migrate --from-boltdb /var/lib/semaphore/database.boltdb --config /etc/semaphore/config-postgres.json
Postgres semaphore_user@localhost semaphore
File is too big  11936063488
Starting migration...
Exporting User                                 100%
Exporting Project                              100%
Exporting Task                                 100%
Exporting Event                                100%
Exporting TaskOutput                           100%
Exporting Integration                          100%
Exporting Option                               100%
Exporting Runner                               100%
Exporting AccessKey                            100%
Exporting Environment                          100%
Exporting Repository                           100%
Exporting View                                 100%
Exporting IntegrationAlias                     100%
Exporting Template                             100%
Exporting TemplateVault                        100%
Exporting TemplateRole                         100%
Exporting SecretStorage                        100%
Exporting Inventory                            100%
Exporting Schedule                             100%
Exporting ProjectUser                          100%
Exporting IntegrationMatcher                   100%
Exporting Role                                 100%
Exporting IntegrationExtractValue              100%
Importing Project                              100%
Importing SecretStorage                        100%
Importing User                                 0%
Migration failed: failed to import User: pq: duplicate key value violates unique constraint "user_username_key" (23505)

The migration configuration:

{
  "bolt": {
    "host": "/var/lib/semaphore/database.boltdb"
  },
  "dialect": "postgres",
  "cookie_hash": "[redacted]",
  "cookie_encryption": "[redacted]",
  "access_key_encryption": "[redacted]",
  "web_host": "https://semaphore.[redacted]",
  "postgres": {
          "host": "localhost",
          "name": "semaphore",
          "user": "semaphore_user",
          "pass": "[redacted]"
  }
}

In the past I installed admin and my own account helio with the same email. But it has been changed. I even tried to delete user admin, but I couldn't pass the error.

Also the database is quite large, 11 GB at the current time.

Impact

Other

Installation method

Binary

Database

BoltDB

Browser

Chrome

Semaphore Version

2.18.2-db3ac46-1778154092

Ansible Version

Logs & errors

No response

Manual installation - system information

No response

Configuration

No response

Additional information

No response

[helioloureiro]

I made a little patch on cli that did the trick:

diff --git a/services/export/User.go b/services/export/User.go
index a6a01e08..1a5d82e9 100644
--- a/services/export/User.go
+++ b/services/export/User.go
@@ -1,6 +1,8 @@
 package export
 
 import (
+       "fmt"
+
        "github.com/semaphoreui/semaphore/db"
 )
 
@@ -24,7 +26,8 @@ func (a *UserExporter) restore(store db.Store, exporter DataExporter, progress P
        if a.MergeExisting {
                users, err := store.GetUsers(db.RetrieveQueryParams{})
                if err != nil {
-                       return err
+                       fmt.Println(err)
+                       return nil
                }
                for _, user := range users {
                        userMap[user.Username] = &user
@@ -41,13 +44,16 @@ func (a *UserExporter) restore(store db.Store, exporter DataExporter, progress P
                } else {
                        obj, err = store.ImportUser(db.UserWithPwd{Pwd: old.Password, User: old})
                        if err != nil {
-                               return err
+                               fmt.Println(err)
+                               //return err
+                               continue
                        }
                }
 
                err = exporter.mapKeys(a.getName(), GlobalScope, old.GetDbKey(), obj.GetDbKey())
                if err != nil {
-                       return err
+                       fmt.Println(err)
+                       //return err
                }
        }
 

Then:

❯ cd cli/
❯ go build -o semaphore-cli main.go
❯ ./semaphore-cli version
undefined-00000000-

Not pretty but good enough to go.

Where I used fmt.Println( ) could be a more usefull error message. It continued forward and imported most of the things to get it working.

The result:

❯ /home/helio/semaphore-cli migrate --from-boltdb /var/lib/semaphore/database.boltdb --config /etc/semaphore/config-postgres.json --merge-existing-users
Postgres semaphore_user@localhost semaphore
File is too big  11936063488
Starting migration...
Exporting User                                 100%
Exporting Project                              100%
Exporting Inventory                            100%
Exporting Integration                          100%
Exporting IntegrationExtractValue              100%
Exporting IntegrationAlias                     100%
Exporting Task                                 100%
Exporting TaskOutput                           100%
Exporting Template                             100%
Exporting TemplateRole                         100%
Exporting AccessKey                            100%
Exporting SecretStorage                        100%
Exporting View                                 100%
Exporting IntegrationMatcher                   100%
Exporting TemplateVault                        100%
Exporting Role                                 100%
Exporting Schedule                             100%
Exporting ProjectUser                          100%
Exporting Option                               100%
Exporting Repository                           100%
Exporting Event                                100%
Exporting Runner                               100%
Exporting Environment                          100%
Importing Project                              100%
Importing User                                 0%pq: duplicate key value violates unique constraint "user_email_key" (23505)
Importing User                                 100%
Importing SecretStorage                        100%
Importing Environment                          100%
Importing AccessKey                            100%
Importing Repository                           100%
Importing Inventory                            100%
Importing View                                 100%
Importing Template                             100%
Importing Schedule                             100%
Importing ProjectUser                          100%
    Errors: 6
Importing Integration                          100%
Importing Task                                 100%
    Errors: 821
Importing Option                               100%
Importing Event                                100%
    Errors: 1765
Importing Runner                               100%
Importing IntegrationExtractValue              100%
Importing IntegrationAlias                     100%
Importing TaskOutput                           100%
Importing Role                                 100%
Importing TemplateRole                         100%
Importing IntegrationMatcher                   100%
Importing TemplateVault                        100%
Migration finished successfully.