qovery / engine Goto Github PK
View Code? Open in Web Editor NEWThe Orchestration Engine To Deliver Self-Service Infrastructure ⚡️
Home Page: https://www.qovery.com
License: GNU General Public License v3.0
The Orchestration Engine To Deliver Self-Service Infrastructure ⚡️
Home Page: https://www.qovery.com
License: GNU General Public License v3.0
The readme says that you support AWS, Digital Ocean, Azure, GCP and Scaleway. But as I can see it only AWS and Digital Ocean are supported currently? Why are the others also listed under that heading?
Same with Build Platforms, Container registries and Monitoring services.
This is really confusing when you try to understand if this piece of software is something I'd be interested in using.
I wanted to create a database in Qovery. I grabbed the db_url
from the UI.
DB_URL = "postgresql://postgres:_Ij=de-6S8B$avgXtFN?@[email protected]:5432/postgres"
(Don't worry, this database does not exist in Qovery anymore.)
I'm getting an error.
File "/home/harshitsinghai/.pyenv/versions/3.9.5/lib/python3.9/urllib/parse.py", line 178, in port
raise ValueError(message) from None
ValueError: Port could not be cast to integer value as '_Ij=de-6S8B$avgXtFN'
I deleted the database, created a new one, and tried again with the new db_url. But I'm getting the same error.
I don't think using postgres
as the default DB username is a good idea. It will be confused by many URL parsers.
postgresql://postgres:U8T%2T#YjO#-ep-MjGBcJ_
this string might be creating confusion for lots of parsers.
Either give the user the option to change the default username or use something more generic.
Code Snippet
QOVERY_URL = "postgresql://postgres:U8T%2T#YjO#-ep-MjGBcJ_@!fgczi!%[email protected]:5432/postgres"
database = databases.Database(QOVERY_URL)
metadata = sqlalchemy.MetaData()
engine = sqlalchemy.create_engine(QOVERY_URL)
Hi 👋
This project uses Heroku's heroku/builder-classic:22
CNB builder image:
engine/src/build_platform/local_docker.rs
Lines 30 to 31 in 056c195
We have deprecated this builder image as of heroku/cnb-builder-images#429.
Please migrate to heroku/builder:22
or heroku/builder:20
, to continue to receive security updates, and to avoid disruption when the build time deprecation warning is upgraded to an error in the future.
More information on the difference between the various builder images can be seen here:
https://github.com/heroku/cnb-builder-images#heroku-cnb-builder-images
The repo URL for the builder images has also changed, see #739.
After reading the code, there is a lot of files that are doing command wrapping by doing shell exec behind the scene.
While doing shell exec is OK as reimplementing everything of what binaries are doing is
A big chunk of the code is riddled with system exec with its own had hoc way of parsing output, calling convention and doing the same kind of operation.
To centralize, standardize and isolate those system-exec, consider moving from raw Command::new
to a simple interpreter.
It will have the benefit of :
Command::new
to creep more into the codePlease find below an example of working interpreter. It can still be improved (timeout can be added, be safer) but provide enough flexibility IMHO.
// Operations available for our shell support
pub enum Shell<'a> {
ExecCmd(&'a str, &'a str),
ExecCmdWithRetry(&'a str, &'a str, u32),
EnsureCmd(&'a str),
SetEnv(&'a str, &'a str),
SetEnvs(&'a Vec<(&'a str, &'a str)>),
}
// Standardized output for all our shell command, a script output is just a Vec<ShellOutput>
#[derive(Deserialize, Debug)]
pub struct ShellOutput {
name: String,
command: String,
error: String,
values: Vec<String>
}
// Snippet of shell script that exec a command and format it's output to a json file "ret.json"
pub fn cmd_output_to_json(name: &str, cmd: &str) -> String {
format!(r#"
##################
# {0}
##################
{1} > ret.stdout 2> ret.stderr
cmd_exit_code=$?
echo '{{ "name": "{0}",' >> ret.json
echo ' "command": "{1}",' >> ret.json
echo ' "error": "'$(cat ret.stderr | tr "\"" "'")'",' >> ret.json
echo ' "values": ['$(cat ret.stdout | xargs -n1 -I R echo '"R"' | paste -sd ',')']' >> ret.json
echo '}}' >> ret.json
echo ',' >> ret.json
[[ $cmd_exit_code -eq 0 ]] || exit 1
"#, name, cmd)
}
// Exec a command with nb max retry
pub fn retry(name: &str, cmd: &str, max_retry: u32) -> String {
format!(r#"
##################
# RETRY {2} - {0}
##################
COUNTER=0
cmd_exit_code=0
while [ $COUNTER -lt {2} ]; do
let COUNTER=COUNTER+1
{1} > ret.stdout 2> ret.stderr
cmd_exit_code=$?
[[ $cmd_exit_code -eq 0 ]] && break
sleep 2
done
echo '{{ "name": "{0}",' >> ret.json
echo ' "command": "{1}",' >> ret.json
echo ' "error": "'$(cat ret.stderr | tr "\"" "'")'",' >> ret.json
echo ' "values": ['$(cat ret.stdout | xargs -n1 -I R echo '"R"' | paste -sd ',')']' >> ret.json
echo '}}' >> ret.json
echo ',' >> ret.json
[[ $cmd_exit_code -eq 0 ]] || exit 1
"#, name, cmd, max_retry)
}
// Transform operation/opcode of our shell into real sh script
fn eval_opcode(action: &Shell, builder: &mut String) {
match action {
Shell::ExecCmd(name, args)=> {
builder.push_str(cmd_output_to_json(name, args).as_str());
},
Shell::SetEnv(name, value) => {
builder.push_str(format!("export {}=\"{}\"\n", name, value).as_str());
}
Shell::SetEnvs(envs ) => {
for (name, value) in envs.iter() {
builder.push_str(format!("export {}=\"{}\"\n", name, value).as_str());
}
}
Shell::ExecCmdWithRetry(name, cmd, max_retry) => {
builder.push_str(retry(name, cmd, *max_retry).as_str());
}
Shell::EnsureCmd(cmd) => {
eval_opcode(&Shell::ExecCmd(format!("ensure command {}", cmd).as_str(),
format!("command -v {}", cmd).as_str()),
builder)
}
};
}
// Generate a complete/executable shell script from our Shell Opcode
pub fn generate_shell(actions: &Vec<Shell>) -> String {
let mut builder = String::with_capacity(1024);
builder.push_str("#!/bin/sh
echo '[' >> ret.json
trap \"rm ret.stdout; rm ret.stderr; sed -i '$ d' ret.json ; echo ']' >> ret.json\" EXIT
");
for cmd in vec![EnsureCmd("cat"), EnsureCmd("sed"), EnsureCmd("paste")] {
eval_opcode(&cmd, &mut builder);
}
for cmd in actions {
eval_opcode(&cmd, &mut builder);
}
builder
}
// Execute our shell script
// 1. Generate the shell script and write it into a temporary folder
// 2. Execute the script with a Command/system exec
// 3. Use serde to parse standard json and extract its value
pub fn exec_shell<P: FromStr>(actions: &Vec<Shell>) -> Result<P, SimpleError>
{
use std::fs::File;
use tempfile::tempdir;
// Generate and write our shell script into a tmp dir (cleaned when variable is droped)
let dir = tempdir()?;
let mut script_path = dir.path().join("run.sh");
File::create(&script_path)?
.write_all(generate_shell(&actions).into_bytes().as_slice());
// Exec our script
let exit = Command::new("sh")
.arg(&script_path)
.current_dir(dir.path())
.output()?;
// Parse ret.json to extract value of the commands
let return_file = File::open(dir.path().join("ret.json"))?;
let results: Vec<ShellOutput> = serde_json::from_reader(BufReader::new(return_file))?;
// Check run success when empty json
// Can be turned into None/Option to simplify
if exit.status.success() && results.is_empty() {
return P::from_str("").or_else(|err| {
Err(SimpleError{
kind: SimpleErrorKind::Other,
message: Some(format!("Cannot parse string to correct format"))
})
});
}
if !exit.status.success() && results.is_empty() {
return Err(SimpleError{
kind: SimpleErrorKind::Command(exit.status),
message: Some(format!("Error executing command: {:?}", exit.stderr))
});
}
// Standard case
for result in results.iter().rev() {
if !result.error.is_empty() {
return Err(SimpleError{
kind: SimpleErrorKind::Command(exit.status),
message: Some(format!("Error executing command: {:?}", result))
});
}
}
P::from_str(results.last().unwrap().values.join("").as_str()).or_else( |err|
Err(SimpleError{
kind: SimpleErrorKind::Other,
message: Some(format!("Cannot parse string to correct format"))
})
)
}
#[cfg(test)]
mod tests {
use crate::cmd::kubectl::Shell::{ExecCmd, ExecCmdWithRetry, EnsureCmd, SetEnv};
use crate::cmd::kubectl::{generate_shell, exec_shell};
#[test]
fn test_generate_shell() {
let actions = vec![
EnsureCmd("kubectl"), EnsureCmd("docker"),
SetEnv("KUBECONFIG", "/home/erebe/.kube/config"),
ExecCmd("get pod number of restart", "kubectl get pods -o=custom-columns=:.status.containerStatuses..restartCount --no-headers=true"),
ExecCmdWithRetry("list lol", "kubectl get lol", 3),
];
//let script = generate_shell(&actions);
//println!("{}", script);
let ret = exec_shell::<String>(&actions);
println!("{:?}", ret);
}
}
It generates script like below
#!/bin/sh
echo '[' >> ret.json
trap "rm ret.stdout; rm ret.stderr; sed -i '$ d' ret.json ; echo ']' >> ret.json" EXIT
##################
# ensure command cat
##################
command -v cat > ret.stdout 2> ret.stderr
cmd_exit_code=$?
echo '{ "name": "ensure command cat",' >> ret.json
echo ' "command": "command -v cat",' >> ret.json
echo ' "error": "'$(cat ret.stderr | tr "\"" "'")'",' >> ret.json
echo ' "values": ['$(cat ret.stdout | xargs -n1 -I R echo '"R"' | paste -sd ',')']' >> ret.json
echo '}' >> ret.json
echo ',' >> ret.json
[[ $cmd_exit_code -eq 0 ]] || exit 1
...
With ret.json file looking like
[
{ "name": "ensure command cat",
"command": "command -v cat",
"error": "",
"values": ["/usr/bin/cat"]
}
,
{ "name": "ensure command sed",
"command": "command -v sed",
"error": "",
"values": ["/usr/bin/sed"]
}
,
{ "name": "ensure command paste",
"command": "command -v paste",
"error": "",
"values": ["/usr/bin/paste"]
}
,
{ "name": "ensure command kubectl",
"command": "command -v kubectl",
"error": "",
"values": ["/usr/bin/kubectl"]
}
,
{ "name": "ensure command docker",
"command": "command -v docker",
"error": "",
"values": ["/usr/bin/docker"]
}
,
{ "name": "get pod number of restart",
"command": "kubectl get pods -o=custom-columns=:.status.containerStatuses..restartCount --no-headers=true",
"error": "",
"values": ["0","0","0","0","0","1","0","0"]
}
,
{ "name": "list lol",
"command": "kubectl get lol",
"error": "error: the server doesn't have a resource type 'lol'",
"values": []
}
]
As an example, 2 kubectl commands rewrote with this simple interpreter
pub fn kubectl_exec_get_number_of_restart(
kubernetes_config: &str,
namespace: &str,
podname: &str,
envs: &Vec<(&str, &str)>,
) -> Result<String, SimpleError> {
let cmd_str = format!("get pods {} -n {} -o=custom-columns=:.status.containerStatuses..restartCount --no-headers=true", podname, namespace);
let cmd = vec![
EnsureCmd("kubectl"),
SetEnv(KUBECONFIG, kubernetes_config),
SetEnvs(&envs),
ExecCmd("pod nb of restart", cmd_str.as_str())
];
exec_shell(&cmd)
}
pub fn kubectl_exec_get_external_ingress_hostnamel(
kubernetes_config: &str,
namespace: &str,
selector: &str,
envs: &Vec<(&str, &str)>,
) -> Result<String, SimpleError>
{
let cmd_str = format!("get svc -o json -n {} -l {}", namespace, selector);
let cmd = vec![
EnsureCmd("kubectl"),
SetEnv(KUBECONFIG, kubernetes_config),
SetEnvs(&envs),
ExecCmd("get external ingress hostname", cmd_str.as_str())
];
exec_shell(&cmd)
}
// Do the same for terraform, docker, etc...
Issues information
Your issue
The list of repositories fetched are not correct with multiple organizations.
Describe here your issue
To prevent duplication and maybe misunderstood please refer to the following thread:
https://community.qovery.com/t/select-a-repo-dont-list-the-repositorie-i-want-to-use/140
I'm not sure if it's the right place to open this issue so feel free to close it or move it to another repository.
Thanks again for the fast reply and support!
this space was intentionally left blank.
Is there any appetite here to work on a C API for Terraform?
Naturally this would then be easy to wrap in higher level languages like Rust; once it's in C.
Really want to avoid using the CLI or Go interfaces, and properly integrate, e.g., into a single binary or dynamic whatever cloud library files are in search directory will be used.
Just found your project, and I assume this would be of interest to you too.
(Happy to help out)
https://discuss.hashicorp.com/t/c-api-to-terraform-providers/14195
First, congrats for this awesome project. I'm triying to create a pilot project with this tool but I've found that kube
dependency seems broken.
Currently, this project depends on kube
version 0.86.0
but it seems yanked in crate.io.
If I tried to build it locally or use it as lib, it seems that it is not possible to compile the project:
cargo build
output:
error: failed to select a version for the requirement
kube = "^0.86.0"
candidate versions found which didn't match: 0.88.1, 0.88.0, 0.87.2, ...
location searched: crates.io index
required by packageqovery-engine v0.1.0 (/home/xxx/qovery/engine)
perhaps a crate was updated and forgotten to be re-vendored?
EDIT: it seems that 0.87.x
works fine.
In my experience Huawei cloud is pretty similar to S3 (though occasionally some things are just ever so slightly different). It would be nice if huawei cloud could be tested and added to the readme if it works. Just because it's probably the most popular cloud provider in china
In the readme the vote link for the Azure cloud (https://github.com/Qovery/engine/blob/dev/README.md?plain=1#L45) is down. It leads to https://hub.qovery.com/docs/using-qovery/configuration/cloud-service-provider/azure/ which throws a 404
It seems that Qovery is heavily inspired by heroku among other things.
I was wondering if Qovery already has a feature similar to heroku-button?
And if not, that it would be considered as a new feature.
We (Heroku) released heroku/builder:24
, our new CNB builder image version based upon Ubuntu 24.04 LTS, in June:
heroku/buildpacks#12
https://github.com/heroku/cnb-builder-images#available-images
Currently this project's default CNB builder image is heroku/builder:22
:
engine/src/build_platform/local_docker.rs
Lines 31 to 32 in f11361b
It would be great to update the default CNB builder image to heroku/builder:24
.
The heroku/builder:24
builder is now a multi-architecture image (AMD64 + ARM64), and offers smaller run image sizes thanks to dropping unnecessary build-time dependencies from the run image. The list of included buildpacks (and their versions) is identical between heroku/builder:22
and heroku/builder:24
.
For more information on the differences between the two images, see:
https://devcenter.heroku.com/articles/heroku-24-stack#what-s-new
https://devcenter.heroku.com/articles/heroku-24-stack#upgrade-notes
https://devcenter.heroku.com/articles/stack-packages
Hi,
I know, it's a goal of the project to get rid of using command line interfaces.
After reading issue #25, I did some research, and I thought it would be useful to start this issue to collect relevant information, and to track progress.
For kubectl
u/garypen proposed kube.rs and k8s-openapi.
For the other tools: Bindings most likely also would be useful to the ecosystems of other programming languages as well. Maybe a solution that's based on gRPC, WASM, or similar would make it easier to secure support from the maintainers of these tools?
Basically, there would need to be a definition of the API (i.e. in JSON) and a gRPC server (or similar). New clients then could automatically generate the native API from the protocol definition (which would be almost trivial in Rust with serde
).
Go has experimental WASM support.
Terraform already has a gRPC server for plugins, but I'm not sure if it could be used to implement a native API for what the command line tool does.
Another approach would be to use different tools, that have better APIs.
Here is a list of alternatives to Terraform, for example:
https://en.wikipedia.org/wiki/Infrastructure_as_code#Tools
Pulumi looks promising. They have already an SDK for TypeScript, JavaScript, Python, and Go, which they seem to have implemented via a gRPC server.
Currently, no support for Rust is planned, but community contributed support would be welcome.
I found the following resources that show how to implement an SDK for a new language:
What do you guys think?
I'd love to use Rust for my Infrastructure as Code needs, but using command line interfaces internally is a dealbreaker for me (and most likely for many other people, as well).
Currently there is Scaleaway support in Qovery, a similar provider who is gaining popularity is Hetzner. So Hetzner Cloud would be an awesome addition.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.