Duplicate entry for key about short-url

[ash-jc-allen]

Hey @prattcmp! Sorry you're having issues with the package. Please could you give me some more information for a bit of context? 🙂

• I've heard of this happening a couple of times when there's been a race condition with queued jobs and that sort of thing. But I'm guessing that because you're getting the error in your browser that you're not using the queue to create the short URLs?
• If possible, could you please show me the code you're using to create the short URLs?
• What key length are you using for generating the keys?
• How many URLs do you currently have in the short_urls table?

[prattcmp]

1. Correct. This isn't happening in queues.
2. Here's the code:
3. Key length is 8
4. 9093 in table

[ash-jc-allen]

Thanks for the info! As far as I'm aware, the hashids package I'm using under the hood does a pretty good job at avoiding collisions as long as it's given some good seed data. So I'm wondering if there's an issue with how I'm grabbing this seed data.

The KeyGenerator grabs the ID of the last inserted short URL like this:

protected function getLastInsertedID(): int
{
    if ($lastInserted = ShortURL::latest()->select('id')->first()) {
        return $lastInserted->id;
    }

    return 0;
}

I'm wondering if there might be an issue with the way I'm grabbing this. Could you please tell me:

• The ID of the short URL that already exists with the key MVwnQpez?
• How many short URLs have been created (and still exist in the DB) since the MVwnQpez was created?

Hopefully we'll be able to get to the bottom of it 🙂

[prattcmp]

Thanks for the help so far!

It was key 8861. We're currently on 9093. So that's about 232 records between collisions. Seems statistically unlikely.

Are you using this library?

I don't think you should be changing the seed on the fly. There are about 136 trillion permutations of those 8 characters, regardless of what the "seed" (salt) is. From hashids:

There are no collisions because the method is based on integer to hex conversion. As long as you don't change constructor arguments midway, the generated output will stay unique to your salt.

The output is unique only to the salt. If you change the salt, you lose uniqueness. Regardless, can you add a try/catch wrapper to the url builder so if it does collide, it retries?

[ash-jc-allen]

Yeah that's the package I'm using 🙂

Sorry, it's probably my poor choice of words, we're not changing the salt after instantiating the Hashids object. We create it like this:

public function __construct(Hashids $hashids = null)
{
    $this->hashids = $hashids ?: new Hashids(config('short-url.key_salt'), config('short-url.key_length'), config('short-url.alphabet'));
}

We then generate the key like so:

public function generateRandom(): string
{
    $ID = $this->getLastInsertedID();

    do {
        $ID++;
        $key = $this->hashids->encode($ID);
    } while (ShortURL::where('url_key', $key)->exists());

    return $key;
}

Thinking about it, I'm not quite sure how duplicates are managing to be created because they should be caught and avoided in the ShortURL::where('url_key', $key)->exists() part 🤔

Off the top of my head, I can't think of any obvious reason that it would be getting through this check. I'll take a look later tonight and see if I can figure out any potential causes for the issue. But if you manage to figure it out before then, I'd be interested to see what the issue was! 😄

[prattcmp]

I'm honestly not sure what's going on. When I wrap in a try/catch with retries, I get repeated collisions. Is there a value being cached somehow?

[ash-jc-allen]

Hey @prattcmp! I've been having a bit of a tinker and I can't seem to reproduce this error. I tried creating 10k short URLs and I couldn't seem to cause any collisions 🤔

Would it be possible for you to try and reproduce the bug in a fresh Laravel project, please? If we can reproduce it in a separate repo, it'll help us to rule out that it's not something specific to your project.

[prattcmp]

Hey @prattcmp! I've been having a bit of a tinker and I can't seem to reproduce this error. I tried creating 10k short URLs and I couldn't seem to cause any collisions 🤔

Would it be possible for you to try and reproduce the bug in a fresh Laravel project, please? If we can reproduce it in a separate repo, it'll help us to rule out that it's not something specific to your project.

Did you try doing this in a while loop with no sleep? I don't see how my project could be causing the collision. None of my code touches your database table or columns. I did not modify your code at all.

We are getting thousands of requests per hour, and sometimes up to 100/min.

[ash-jc-allen]

Yeah, I used a while loop without any sleep.

I'm still not entirely sure what could be causing this and I've not been able to reproduce the bug. Without a demo project or some steps to reproduce, I don't think I'll be able to do much more digging on this one, sorry. Is it possible for you to provide me with either of these? 🙂

[mah3uz]

Hey @ash-jc-allen Many many Thanks for your great work on the package.
I am also getting same issues time to time.
In my project I have key length: 5 and 27700+ record.

Also I am now getting

N+1 Query
select exists(select * from 'short_urls' where 'url_key' = ?) as 'exists'

I am not an experienced developer, So if you could direct me in the right direction it would be helpful.
Should I increase key length?

[ash-jc-allen]

Hey @mah3uz! That's pretty handy to know that you're running into the same problem because it might make it easier for me to spot what's causing it.

Do you have any steps to reproduce the issue, or have a fresh Laravel project that has this issue?

I've still been struggling to cause this to happen, so anything like that would be super helpful 🙂

[ash-jc-allen]

Hey! I'm going to close this issue because I've still not been able to reproduce the error. But if anyone manages to create a demo repo containing the issue, feel free to open another issue and I'll take a look 🙂

[albertovincenzi]

Hello,
I'm facing the same issue. I have a 29008 records and on 10 runs, I got 3127 attempts to find a non-existing key.
What do you think if instead of generating the id starting from last insert id, you use something like microtime() to replace $ID variable in generateRandom method?

[ash-jc-allen]

Hey @albertovincenzi! I think over the next couple of months, I'm going to try and introduce an interface for the key generation class. I'm hoping that by doing this, it means that people can switch over to their own implementation for generating the keys if the package's implementation doesn't do the job anymore 🙂