Viable Hierarchical Storage Management products on market?

A lot of scientific and geographic data (mostly files on Windows and Linux file servers/filesystems) that needs to be retained effectively forever. Currently scattered on disk storage, random external media (offline) and custom tapes (offline, needs admin intervention to access).

Plan is to make it all accessible semi-online for end-users. As workflows are file-based, it would make sense to archive seldomly used files to tape and leave behind stubs for automatic recall if files are touched (or some other self-service for recall). Existing external media and tape would be migrated to this HSM system.

I know that there used to be more HSM products on the market, but currently I can only find IBM Spectrum Protect for Space Management and Veritas Enterprise Vault (no Linux support on first glance).

Any suggestions on HSM products? Alternative solutions?

👍︎ 7
💬︎
👤︎ u/DonZoomik
📅︎ Dec 29 2020
🚨︎ report
Transparent Hierarchical Storage Management with Apache Kudu and Impala kudu.apache.org/2019/03/0…
👍︎ 2
💬︎
👤︎ u/granthenke
📅︎ Mar 20 2019
🚨︎ report
Archivy v1.6 - Extensible Self-Hosted Knowledge Management, allowing integration of web content and both hierarchical and tag organizations archivy.github.io
👍︎ 42
💬︎
📅︎ Dec 15 2021
🚨︎ report
🚀 Burdy 2.0 Release - Preview Editor, Backup Management, Hierarchical Posts!

Dear community,

We are proud to present Burdy 2.0 - a new iteration of headless CMS that will simplify the management of your Next.js content!

The Burdy 2.0

We've been working closely with our enterprise partner for the last couple of months on new features and improvements to the Burdy platform. As a result, Burdy is now easier to use and offers simpler management of your content. One of the key points that were addressed and improved is omnichannel capability and backup management.

You can read more about the changes in our blog.

In parallel, we've been working on a new website that will make use of the new features, and make it easier for content writers to manage the website without support from the developers. We will be open-sourcing the template (our website) in the following days, and you will be able to create your own website within minutes.

Our website is built in Next.js, deployed to Vercel (Burdy is on DigitalOcean) and we've connected the Preview Mode with it.

Here's a video that covers the usage of the new features.

The Burdy 2.0

We will be also providing more clarity on how to deploy and use Burdy, by improving our documentation.

We would appreciate any feedback, suggestions, or questions you may have about the Burdy 2.0! If you'd like to support us, please leave us a star on our GitHub.

Thank you!

👍︎ 14
💬︎
👤︎ u/team-burdy
📅︎ Nov 11 2021
🚨︎ report
Scaling Kubernetes Tenant Management with Hierarchical Namespaces Controller engineering.mercari.com/b…
👍︎ 5
💬︎
👤︎ u/tcnksm4481
📅︎ Oct 05 2021
🚨︎ report
Devtron - Opensource Kubernetes delivery Workflow for CI/CD with security features like Granular Hierarchical Security policy Management, Automatic Vulnerability management during CI as well as within cluster. github.com/devtron-labs/d…
👍︎ 11
💬︎
👤︎ u/pghildiy
📅︎ Jul 28 2021
🚨︎ report
Given we're such social creatures, evolution selects for things psychologically like hierarchical thinking, status, reputation management. So...gossip has as much of a function as auditory processing, vision
👍︎ 10
💬︎
👤︎ u/meta_mamet
📅︎ May 23 2021
🚨︎ report
[Socialists]If the workers at a co-op are discovered to have instituted a hierarchical management structure among themselves, how would this be corrected?

In other words, who would be responsible for enforcing workplace democracy, and what means would they have at their disposal?

👍︎ 101
💬︎
👤︎ u/Impacatus
📅︎ Apr 03 2020
🚨︎ report
Unveiling Intrinsic Potassium Storage Behaviors of Hierarchical Nano Bi@N‐Doped Carbon Nanocages Framework via In Situ Characterizations

A hierarchical nano Bi@N‐doped carbon nanocage framework with an interior void space was smartly fabricated as competitive anode for K‐ion batteries. Its intrinsic potassium storage behaviors are unveiled via comprehensive in situ characterizations.

Abstract

Metallic bismuth has drawn attention as a promising alloying anode for advanced potassium ion batteries (PIBs). However, serious volume expansion/electrode pulverization and sluggish kinetics always lead to its inferior cycling and rate properties for practical applications. Therefore, advanced Bi‐based anodes via structural/compositional optimization and sur‐/interface design are needed. Herein, we develop a bottom‐up avenue to fabricate nanoscale Bi encapsulated in a 3D N‐doped carbon nanocages (Bi@N‐CNCs) framework with a void space by using a novel Bi‐based metal‐organic framework as the precursor. With elaborate regulation in annealing temperatures, the optimized Bi@N‐CNCs electrode exhibits large reversible capacities and long‐duration cyclic stability at high rates when evaluated as competitive anodes for PIBs. Insights into the intrinsic K+‐storage processes of the Bi@N‐CNCs anode are put forward from comprehensive in situ characterizations.

https://ift.tt/2NorvGb

👍︎ 2
💬︎
📅︎ Mar 17 2021
🚨︎ report
I REALLY wish that the storage management screen showed if games are X optimised

It would make deciding which games to transfer to an external HDD much easier than jumping back and forth between different screens and trying to remember if a game should be on the internal / Seagate drives.

👍︎ 113
💬︎
👤︎ u/Farley_UK
📅︎ Jan 04 2022
🚨︎ report
Management requested small storage devices for crochet hooks and the like.
👍︎ 640
💬︎
📅︎ Dec 25 2021
🚨︎ report
Polymer Stabilized Droplet Templating towards Tunable Hierarchical Porosity in Single Crystalline Na3V2(PO4)3 for Enhanced Sodium‐Ion Storage

A facile polymer stabilized droplet template strategy allows the synthesis of porous single crystal structured NVP with tunable pore structure. The resultant hierarchically meso/macropores NVP demonstrates superior sodium storage performances, because its porous single crystal structure shortens ion diffusion distance and provides large electrode–electrolyte contact area, greatly facilitating fast ionic transport.

Abstract

Na3V2(PO4)3 (NVP) is regarded as a potential cathode material for sodium‐ion batteries, whereas, its performance is usually limited by inherent low electronic conductivity and dense bulk structure. Herein, we develop a facile polymer stabilized droplet template strategy to synthesize porous single crystal structured NVP. The pore structures (macrostructures, hierarchically meso/macrostructures, and mesostructures), pore sizes (5–2000 nm), and specific surface areas (26–158 m2 g−1) of the samples can be readily controlled by tuning the sizes of droplet templates. The resultant hierarchically meso/macropores NVP demonstrates superior sodium storage performances, because its porous single crystal structure owns solid–liquid Na+ transmission mode, shortens ion diffusion distance and provides large electrode–electrolyte contact area, greatly facilitating fast ionic transport. We believe the presented method will supply a novel avenue to prepare porous single crystal structured materials for anticipative applications.

https://ift.tt/3bVXBm4

👍︎ 2
💬︎
📅︎ Apr 24 2021
🚨︎ report
How much help can Hierarchical Deterministic trees of keys help with key management for non-expert users?

I've recently been made aware of BIP32, which was invented to make "Hierarchical Deterministic Wallets" (HD wallets) in BitCoin. I was wondering what uses this could have outside crypto currencies most notably for your "regular" cypherpunk using tools like GPG or Age to communicate with their web of trust.

A deterministic tree of key pairs basically works like this: you start with a root key pair, that must be generated once and never lost or compromised. Then you generate sub-keys by hashing that root key with an easily remembered index). If a sub-key is lost, it can be re-generated from the root key. Now, BIP32 has two ways of generating sub keys, each with their own tradeofs.

Note: I'll use the following names from now on:

G         -- Generator of the group (public constant)
a         -- root private key
A = a.G   -- root public key
b         -- child private key
B = b.G   -- child public key.
i         -- public index (each child key has its own unique index)

Hardened keys are generated from the private half of the root key (over-simplified for clarity):

b = KDF(a, i)
B = b.G

Key derivation can't be reversed, so if the child key b happens to be compromised, the root key a is still safe. The advantage of the deterministic generation is that if you lose the child key (you dumped your cell phone, your hard drive fried…) you can re-generate it from the root key, and pretend you never lost it.

Non-hardened keys are generated from the public half of the root key, such that even third parties can generate it:

z = KDF(A, i)
Z = z.G
b = a + z    -- modulo group order
B = A + Z
B = a.G + z.G
B = (a+z).G
B = b.G    

Anyone can generate the public key, but generating the private key requires knowledge of the root private key. As far as I know, this is safe, because breaking this scheme would mean that we have solved the Discrete Logarithm Problem. However, if a non-hardened child key b is compromised, so is the root key. z is public (derived from the public root key), so knowing b easily reveals a:

b = a + z
a = b - z

Unless I'm missing something, this means we should not store non-hardened key pairs less securely than we store the root key itself.

Is there a compelling use case?

I was wondering how useful those could be, compared to a simple hierarchy of certified keys, where child key

... keep reading on reddit ➡

👍︎ 15
💬︎
📅︎ Oct 28 2020
🚨︎ report
Problems with storage management

So I'm about 30 hr in this game, my third colony grew pretty big (for me) about 400ppl, biggest problem, kind of game breaking is about food, basically even if I've plenty of food colonist starve cause they can't get it from the depots, i had to delete the big depots cause they stop working after some time and i have to manually "reset" after it works for 5 or 10 units of food and gets stuck again, I've one big depot that has i think 100 colonists stuck inside and i cant delete it cause it's half full but won't empty.... I somehow fixed the food problem by placing 4 depots for each dome and this works for some time but yesterday before ending my session I've noticed colonists stuck in empty depots.....

Other little thing, but i think it's relative to problems created by food bugs is about concrete, and how it takes way too much time to bring it to construction sites, i have about 150 drones and 7 shuttle hub with 12 each i think the problem is just the huge amount of drones stuck on food depots...

Anyway to solve this? I've noticed posts from years ago about the same but couldn't solve it

👍︎ 5
💬︎
👤︎ u/SoN_FrAnK
📅︎ Jan 06 2022
🚨︎ report
Storage management

I really need pocketcasts to introduce some storage management. I have 273 GB of data in podcasts but no way to drill down to do a selective deletion or even just understand how it's distributed over the different podcasts. Some material will not be available to download anymore, so I can't just do a reset.

Also filters won't tell me how much space the files occupy that I have in the respective filters. Even the simplest of statistics - "how much space do the episodes of a certain podcast occupy?" isn't available as far as I can see.

The only statistics I can find are on the episode level (how many MB for the current episode) or globally in the manage downloads function. But the granularity this offers is comparable to trying to fix a watch with a sledgehammer... apparently I have 297 GB in the unplayed category and there is a delete option. Well thanks, but that doesn't help.

And instead of keeping the filenames original they are renamed according to pocketcast's filing system and all I can do is see the file type or start them up to get the id3 tags.

👍︎ 12
💬︎
👤︎ u/telakin
📅︎ Jan 11 2022
🚨︎ report
Rate my build: ~6TB ram across 4 hosts, ~2.5 petabytes of storage (sorry bout the cable management haha first build)
👍︎ 7k
💬︎
👤︎ u/Ylurpn
📅︎ Aug 13 2021
🚨︎ report
Going beyond the hierarchical file system: a new approach to document storage and retrieval pages.stern.nyu.edu/~marr…
👍︎ 2
💬︎
📅︎ Aug 09 2020
🚨︎ report
Has anybody come up with a decent storage management system for their items?

Look, I could spend some time writing out a few paragraphs about how trash the storage system is. But I'm more curious if anybody has had a decent system work for them?

I saw a great post where somebody recommended putting low-tier cut gems into their different luck sets. I'm thinking more for where people organize their storages across the world.

👍︎ 17
💬︎
📅︎ Dec 11 2021
🚨︎ report
Local storage management

My local storage USB is full. I can only delete old clips one by one on the app, which is very slow since it needs to load each clip before I can delete them. I tried putting the USB into a computer which let me look at the files but it wouldn't allow me to delete them. Is there any way to do this at a reasonable speed? We're talking thousands of clips at this point.

👍︎ 3
💬︎
📅︎ Dec 24 2021
🚨︎ report
DoorDasher encounters evil---"Picking up an order and I heard an employee beg his manager to use the restroom."--- Hierarchical work-places are coercive, abusive, sociopathic, and toxic and this is what the anti-anti-work people refuse to acknowledge or are unable to understand. /r/doordash_drivers/comme…
👍︎ 33
💬︎
📅︎ Oct 01 2021
🚨︎ report
So how do I use my hdd I installed for games on my nitro 5? I got the 1TB seagate barracuda but in system storage it doesn’t say I have it installed, just shows the 237gb that came with the laptop but if I go into the intel storage management it shows both. reddit.com/gallery/s0sic2
👍︎ 3
💬︎
👤︎ u/Xignith
📅︎ Jan 10 2022
🚨︎ report
Tell me you have ADHD without saying you have ADHD. (I’m terrible with email management and finally out of Google storage)
👍︎ 511
💬︎
📅︎ Oct 13 2021
🚨︎ report
Renewable energy skeptics argue that because of their variability, wind and solar cannot be the foundation of a dependable electricity grid. But the expansion of renewables and new methods of energy management and storage can lead to a grid that is reliable and clean e360.yale.edu/features/th…
👍︎ 45
💬︎
👤︎ u/stefeyboy
📅︎ Dec 18 2021
🚨︎ report
Am I stupid, or does this game have a terrible drone/storage management system?

So ive been picking up Surviving Mars again since yesterday. I immediatley noticed that Drones dont pick up stuff from storage places, if it is outside of their hub range, but inside a other hubs range?
How do I make drones pick up materials from one hubs area, to bring it to a other hubs area?

Currently, I have to constantly move my RC Transporter around to pick up simple items to then manually bring them other somewhere else to have stuff being built there. OR I have storages within overlaps of a few Hubs, but that only works for like 2 or 3 hubs, because then the new hubs are way out of range. And I certainly dont want to build storage places on each hubs area edge, do I?

So, am I stupid, or does this game have a terrible management system and AI?

👍︎ 5
💬︎
👤︎ u/TrackLabs
📅︎ Dec 11 2021
🚨︎ report
Light/Electricity Energy Conversion and Storage for a Hierarchical Porous In2S3@CNT/SS Cathode towards a Flexible Li‐CO2 Battery

Battery life on Mars: A photoinduced flexible Li‐CO2 battery with hierarchical, porous, and free‐standing In2S3@CNT/SS as a bifunctional photoelectrode to accelerate both CO2 reduction and evolution is presented. The Li‐CO2 battery achieved a record‐high discharge voltage of 3.14 V (thermodynamic limit: 2.80 V), and an ultra‐low charge voltage of 3.20 V, and a roundtrip efficiency of 98.1 %.

Abstract

A photoinduced flexible Li‐CO2 battery with well‐designed, hierarchical porous, and free‐standing In2S3@CNT/SS (ICS) as a bifunctional photoelectrode to accelerate both the CO2 reduction and evolution reactions (CDRR and CDER) is presented. The photoinduced Li‐CO2 battery achieved a record‐high discharge voltage of 3.14 V, surpassing the thermodynamic limit of 2.80 V, and an ultra‐low charge voltage of 3.20 V, achieving a round trip efficiency of 98.1 %, which is the highest value ever reported (2 by reducing In3+ to In+ under light illumination. The mechanism of the bifunctional light‐assisted process provides insight into photoinduced Li‐CO2 batteries and contributes to resolving the major setbacks of the system.

https://ift.tt/2ZhZaF5

👍︎ 2
💬︎
📅︎ Oct 20 2020
🚨︎ report
Hierarchical storage locations

How to set up hierarchical storage locations like Fridge Shelf 1 Shelf 2

👍︎ 2
💬︎
👤︎ u/Saturo78
📅︎ Sep 20 2020
🚨︎ report
A tool that would be really useful for diamonds and proxy smart contracts is a hardhat plugin that checks that the order of all state variables is correct before doing an upgrade. And for this to work with Diamond Storage and AppStorage state variable management techniques. twitter.com/mudgen/status…
👍︎ 7
💬︎
👤︎ u/mudgen
📅︎ Jan 17 2022
🚨︎ report
MA property management company removed tool chest from owned condo basement citing fire hazard (it was within area limits but an eye sore) then later state they moved it to the electric room and I need to cover $25/day storage fee and need it removed not back to the previous spot. Options?
👍︎ 6
💬︎
👤︎ u/JBean85
📅︎ Dec 20 2021
🚨︎ report
Dual Monitor Setup, took 3 years to completePC Specs: CPU:i5 9400f ,GPU:RTX 3060 ,RAM: 16 gb 3200mhz ,Motherboard: MSI Z390A-PRO ,PSU: 750W DEEPCOOL ,STORAGE: 512Gb M.2 SSD Gen3 & 1TB HDD ,LEFT monitor : MSI 144hz 27 inch RIGHT monitor: LENOVO 240hz 24.5 inch Currently working on wire management
👍︎ 29
💬︎
👤︎ u/HulkSk007
📅︎ Dec 12 2021
🚨︎ report
Steam Client Update, September 9 (visual updates for download page, storage management, and more)

https://steamcommunity.com/games/593110/announcements/detail/2941402621044012222

Steam has released an update that has substantial changes to the download page, storage management popup, library and much more.

This update is significant because it adds much more information to the UI, specifically around data management:

Download page

  • Downloads now display total progress which includes disk allocation information

  • Partial downloads get visual update

  • Type of content (DLC, Workshop, Shaders) is now given in a tooltip

  • Queue can be managed by drag and drop

  • Can now choose to launch game when download is complete through new context menu

  • Patch notes now linked rather than "View News"

Storage Management

  • Library folder UI clearer info for multiple drives

  • Mount read-only drives

  • Allow removing empty library folders

Library Window

  • Clearer dialogue for currently installed content

  • Various touch ups

Fixed lots of bugs, see above link for more information

It seems that steam very rarely makes updates to its UI but this feels like a step in the right direction. How do you feel about the update, what are some other improvements that you'd like to see over time?

👍︎ 898
💬︎
👤︎ u/PhazonJim
📅︎ Sep 11 2021
🚨︎ report
Hierarchical blob structure in storage

Hi, I'm trying to create a hierarchy of blobs in ADLS2 where I have access to the top blob and other people in my org only have access to sub-directories/sub-blobs further down in the hierarchy. However I notice you can't create blobs inside of blobs, so am I thinking this wrong? I can create folders inside blobs but don't think I can grant granular access policies to these blobs. I also don't want to create multiple Blobs/storages since that would require a lot of extra logic in ADF where I'm copying files from one data lake to another.

👍︎ 6
💬︎
📅︎ Jan 14 2020
🚨︎ report
INVENTORY MANAGEMENT GUIDE! Optimize Your Storage System!

Hey again everyone!

I noticed while going through the game, the hardest thing for me was always inventory management. Not only can be anxiety inducing but extremely overwhelming. So I figured what better way to fix that then develop a foundational system to base it on. Therefore, I thought I would share with you all as well :)

I also cover this in more detail with examples and references in a video if you would like to watch it: https://youtu.be/nwu_Ttv8eqU

I based this foundation off a few concepts:

Storage chests, storage sheds, territory standing & control, and trading posts advantage.

Ideal Locations:

What territory to decide?

I recommend using the 3 major economical towns and 2 minor economical ones.

Major:

  • Everfall
  • Windsward
  • Brightwood

Minor:

  • Monarchs Bluff
  • First Light

Now there are 3 factors that I would weigh into deciding the town

  1. Faction Control - if your faction controls then it is easy for you to transfer items across storage sheds. The easiest but most expensive method
  2. Company-control length - What I mean by this is look at how long a company has controlled a certain territory and pay attention to what crafting stations they seem to focus on the most. Some places may put all of their focus into getting Tier 5 refining stations. A compliment to refining reagents & gathered resources. Some may like Tier 5 Forge & Outfitter at all times. A good compliment to armoring related resources. And so forth.
  3. House Location & Farm routes - Try to double down on this. If you have 3 farm routes you go to all the time that are all around Windsward, consider making Windsward a ideal location for dumping resources using the storage shed and/or storage chest.

How to Store:

#1 Spot - Refining Reagents

Sandpaper, Tannin, Solven, Weave, and Flux all when stockpiled, take up ALOT of weight. Look for a location that focus on the refining stations and make that home base for these items. I would recommend combining this with house storage.

#2 Spot - Gearsets/Ammo/Craft Mods

Choosing a neutral location that doesn't benefit you in any other way than to use it as a storage town. Ideally, you would want this faction controlled. But a place with high taxes, no farm routes nearby, and low tier stations is beneficial for a true storage location because that is your only benefit at that point.

#3 Spot - Fishing/Cooking

All of the meats and food items can add up quickl

... keep reading on reddit ➡

👍︎ 33
💬︎
📅︎ Nov 04 2021
🚨︎ report
Point in the right direction for mass storage management

Hi fellas, struggling to understand how to use all my drives in my set up. So built a machine, planned to use ESXi 7, to play with for a little while and then maybe move to proxmox if i cant get a license.

So i have an esxi 6.7 set up (didn't have an intel nic, and i injected the Realtek drivers into the 6.7 builds so no biggie, cant seem to do that with 7 due to some issues ).

So i added a few drives, set them all up as new datastores but now im confused about how i store stuff. I can allocate my vms to whichever data store but how do i do that across multiple drives ?

For example, i have 2 running fine, which is a window 10 vm and a debian one, one does my home cameras and has about 300gb allocated from a 4tb drive, and the same one has my 20gb valhiem server, all works...all good

My third however is my jellyfin media server, its on a debian vm but struggling to work out the storage, as it stands the media will be across 2/3 drives, and i can only allocate one, would i need to allocate each drive im using as an extension to the data store in the storage section or is there a better way to do this ?

I considered using a truenas vm, wondering if im best to set up some other nas like service and then add it as a network drive rather than direct connect in the vm settings.

New to all this, and currently trying to work out VIC to play with a jellyfin container, but i dont know enough yet to jump into that fully whilst just getting my feet wet. Ill do a migration to proxmox in a few months. Which i know will be super fun, but i wanted to get some more exp with esxi, as ill be using that for work after xmas

👍︎ 2
💬︎
📅︎ Dec 20 2021
🚨︎ report
Optimal inventory/storage management! v.redd.it/yeib9uxr0kl71
👍︎ 359
💬︎
👤︎ u/TheJonzu
📅︎ Sep 04 2021
🚨︎ report
Dockerhub vs. Dockstation for storage management?

Pros/Cons of each in the homelab scenario?

I would connect to docker instances from The Cloud^(TM) or my own docker VM's using the above apps.

The big thing I really need from the management UI is simple and persistent storage for container configurations that is handled automagically or at least in an easy way.

If I have a container that is set up perfectly has all configuration and persistent data saved, but needs an update, which app handles the volume re-attachment to the new container?

Edit: Docker Desktop vs. Dockstation, hub is the image repository

👍︎ 2
💬︎
📅︎ Nov 24 2021
🚨︎ report
[P] Confugue: Python package for hierarchical configuration management

I'm releasing Confugue, a hierarchical configuration framework for Python. I wrote it to manage model configurations in my deep learning research, and it's been really useful. TL;DR: Install with pip install confugue and check out the docs or the notebook with PyTorch examples.

Why should I use it?

If your model or algorithm has lots of (hyper)parameters, managing all of them in one place can easily get messy. Confugue makes your life easier by providing a smart wrapper for nested configuration dictionaries (loaded from YAML or JSON files), which automatically fills in class and function parameters with the appropriate values. This eliminates the need for passing these values around or even explicitly referencing them.

Examples of things that work:

  • Arbitrary nesting of the configuration file according to the structure of your code
  • Specifying defaults and additional arguments in the code
  • Omitting a whole section of the configuration file if defaults are provided
  • Using a list of dicts to configure a list of objects with different arguments for each
  • Overriding the type of an object while preserving the arguments passed by the caller
  • Detecting unused configuration values

Example

Here's a quick example using PyTorch. (However, Confugue is equally easy to use with TensorFlow or in non-ML code!) See the Jupyter/Colab notebook for more extensive examples, and the docs for advanced features.

from confugue import configurable, Configuration

@configurable
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = self._cfg['conv1'].configure(nn.Conv2d, in_channels=3)
        self.conv2 = self._cfg['conv2'].configure(nn.Conv2d)
        self.pool = self._cfg['pool'].configure(nn.MaxPool2d)
        self.fc = self._cfg['fc'].configure(nn.Linear, out_features=10)

    def forward(self):
        ...

@configurable
def main(num_epochs=1, *, _cfg):
    net = _cfg['net'].configure(Net)
    optimizer = _cfg['optimizer'].configure(torch.optim.SGD,
... keep reading on reddit ➡

👍︎ 3
💬︎
📅︎ Apr 28 2020
🚨︎ report
Hierarchical storage with data deduplication

Hi Homelab, I am upgrading my workstation and was curious about storage. In the server world, there are many interesting techniques not available to the home user yet.

I am looking to know if single node data deduplication for tiered storage is a thing in the Linux world yet, Google/Bing have failed me.

The setup I want:

  1. pool of ram
  2. 3dxp nvme SSD (with deduplication)
  3. spinny drive

I haven't seen this done yet, but it may be because my terminology is wrong. Any pointers you can share?

EDIT: also, Merry Christmas all.

Edit: The chosen solution for the moment is to just buy a big honking SSD instead of worrying about all these items, when ceph/zfs/other have a turnkey solution, I will do it. I need to balance out my sysadminery with family time. ;)

Thanks again for all the input, sorry if this is anti-climactic.

👍︎ 21
💬︎
📅︎ Dec 25 2018
🚨︎ report
Azure Storage Architecture/Management for Streaming/High throughput?

Hi all,

I am wanting to get other peoples opinions on the storage architecture for landing Streaming data in Azure, taking in to account storage limitations and lifecycle management (Hot to Cold to Archive).

P.S I have only included data lake + delta Lake as the final processed storage to simplify the design. Yes there would be a WH too if needed.

__ My thought processes __

The standard Architecture I see for storing raw streaming for keep long term is to put in to Azure Datalake / Blob storage. Combined with Batch processing and then to Delta Lake for processing, it would look like this:

Batch and stream to data lake for raw, then to Delta lake for processing

However, I recently came across this comment about storing log analytics from azure monitor :

"Don't use an existing storage account that has other, non-monitoring data stored in it to better control access to the data and prevent reaching storage ingress rate limit, failures, and latency. " From https://docs.microsoft.com/en-us/azure/azure-monitor/logs/logs-data-export?tabs=portal.

It's the first time I have seen this and have been thinking about what it means. I have always kind of assumed that there should be only one data lake base on the idea of reducing data silos to one. This idea was first crushed when I saw the delta lake is on it's own (at least based on example I have seen). My take way is that each streaming source should have it's own blob storage/data lake sink. When I think about how this might look with multiple streams of data and life cycle management. I came up with this:

https://preview.redd.it/t41phh6dpv181.png?width=759&format=png&auto=webp&s=ac32cb1423bf6434e7219898c914535893ad480d

The blobs could also be Data lakes, I'm not sure on which is better for this scenario.. I also suspect that The delta lake would also have azure lifecycle management turned on too.

Alternately, I have seen Blobs Storage referenced to as a temporary storage (I forget where). In which case the design could also be:

https://preview.redd.it/mlabqyl3sv181.png?width=916&format=png&auto=webp&s=24f6a262e61b1271e034510651e5d8ba6d25b387

This design weakness is that there is more data movement costs, but it's advantage is having one long term storag

... keep reading on reddit ➡

👍︎ 11
💬︎
👤︎ u/pych_phd
📅︎ Nov 26 2021
🚨︎ report
Unveiling Intrinsic Potassium Storage Behaviors of Hierarchical Nano Bi@N‐Doped Carbon Nanocages Framework via In Situ Characterizations

A hierarchical nano Bi@N‐doped carbon nanocage framework with an interior void space was smartly fabricated as competitive anode for K‐ion batteries. Its intrinsic potassium storage behaviors are unveiled via comprehensive in situ characterizations.

Abstract

Metallic bismuth has drawn attention as a promising alloying anode for advanced potassium ion batteries (PIBs). However, serious volume expansion/electrode pulverization and sluggish kinetics always lead to its inferior cycling and rate properties for practical applications. Therefore, advanced Bi‐based anodes via structural/compositional optimization and sur‐/interface design are needed. Herein, we develop a bottom‐up avenue to fabricate nanoscale Bi encapsulated in a 3D N‐doped carbon nanocages (Bi@N‐CNCs) framework with a void space by using a novel Bi‐based metal‐organic framework as the precursor. With elaborate regulation in annealing temperatures, the optimized Bi@N‐CNCs electrode exhibits large reversible capacities and long‐duration cyclic stability at high rates when evaluated as competitive anodes for PIBs. Insights into the intrinsic K+‐storage processes of the Bi@N‐CNCs anode are put forward from comprehensive in situ characterizations.

https://ift.tt/38KEbhh

👍︎ 2
💬︎
📅︎ Feb 17 2021
🚨︎ report
Polymer Stabilized Droplet Templating towards Tunable Hierarchical Porosity in Single Crystalline Na3V2(PO4)3 for Enhanced Sodium‐Ion Storage

A facile polymer stabilized droplet template strategy allows the synthesis of porous single crystal structured NVP with tunable pore structure. The resultant hierarchically meso/macropores NVP demonstrates superior sodium storage performances, because its porous single crystal structure shortens ion diffusion distance and provides large electrode–electrolyte contact area, greatly facilitating fast ionic transport.

Abstract

Na3V2(PO4)3 (NVP) is regarded as a potential cathode material for sodium‐ion batteries, whereas, its performance is usually limited by inherent low electronic conductivity and dense bulk structure. Herein, we develop a facile polymer stabilized droplet template strategy to synthesize porous single crystal structured NVP. The pore structures (macrostructures, hierarchically meso/macrostructures, and mesostructures), pore sizes (5–2000 nm), and specific surface areas (26–158 m2 g−1) of the samples can be readily controlled by tuning the sizes of droplet templates. The resultant hierarchically meso/macropores NVP demonstrates superior sodium storage performances, because its porous single crystal structure owns solid–liquid Na+ transmission mode, shortens ion diffusion distance and provides large electrode–electrolyte contact area, greatly facilitating fast ionic transport. We believe the presented method will supply a novel avenue to prepare porous single crystal structured materials for anticipative applications.

https://ift.tt/3bVXBm4

👍︎ 2
💬︎
📅︎ Mar 18 2021
🚨︎ report
Unveil Intrinsic Potassium Storage Behaviors of Hierarchical Nano Bi@N‐Doped Carbon Nanocages Framework via In Situ Characterizations

Recently, metallic bismuth has drawn enormous attentions as one promising alloying anode for advanced potassium ion batteries (PIBs), owing to its innate merits. However, serious volume expansion/electrode pulverization and sluggish kinetics always lead to its inferior cycling and rate properties for practical applications. Therefore, it is urgent yet meaningful to construct advanced Bi‐based anodes via structural/compositional optimization and sur‐/interface design. Herein, we first purposefully develop a bottom‐up avenue to efficiently fabricate nanoscale Bi encapsulated in three‐dimensional N‐doped carbon nanocages (Bi@N‐CNCs) framework with a void space by using a novel Bi‐based metal‐organic framework as the precursor. With elaborate regulation in annealing temperatures, the optimized Bi@N‐CNCs electrode exhibits large reversible capacities and long‐duration cyclic stability at high rates when evaluated as competitive anodes for PIBs, benefiting from its unique structure/composition advantages. The in‐depth insights into the intrinsic K+‐storage processes of the Bi@N‐CNCs anode are put forward with comprehensive in situ characterizations. More significantly, the methodology and understanding here hold enormous promise in rational design and versatile synthesis of multi‐component Bi‐based alloys as electrodes towards rechargeable batteries and beyond.

https://ift.tt/38KEbhh

👍︎ 2
💬︎
📅︎ Dec 29 2020
🚨︎ report
Light/Electricity Energy Conversion and Storage for a Hierarchical Porous In2S3@CNT/SS Cathode towards a Flexible Li‐CO2 Battery

Battery life on Mars : A photoinduced flexible Li‐CO2 battery with hierarchical, porous, and free‐standing In2S3@CNT/SS as a bifunctional photoelectrode to accelerate both CO2 reduction and evolution is presented. The Li‐CO2 battery achieved a record‐high discharge voltage of 3.14 V (thermodynamic limit: 2.80 V), and an ultra‐low charge voltage of 3.20 V, and a roundtrip efficiency of 98.1 %.

Abstract

A photoinduced flexible Li‐CO2 battery with well‐designed, hierarchical porous, and free‐standing In2S3@CNT/SS (ICS) as a bifunctional photoelectrode to accelerate both the CO2 reduction and evolution reactions (CDRR and CDER) is presented. The photoinduced Li‐CO2 battery achieved a record‐high discharge voltage of 3.14 V, surpassing the thermodynamic limit of 2.80 V, and an ultra‐low charge voltage of 3.20 V, achieving a round trip efficiency of 98.1 %, which is the highest value ever reported (2 by reducing In3+ to In+ under light illumination. The mechanism of the bifunctional light‐assisted process provides insight into photoinduced Li‐CO2 batteries and contributes to resolving the major setbacks of the system.

https://ift.tt/2ZhZaF5

👍︎ 2
💬︎
📅︎ Jul 01 2020
🚨︎ report
Light/Electricity Energy Conversion and Storage for Hierarchical Porous In2S3@CNT/SS Cathode towards a Flexible Li‐CO2 Battery

The low actual specific capacity, high overpotentials, and poor rate capability due to sluggish kinetics of carbon dioxide reduction reaction (CDRR) and evolution reaction (CDER) hinder practical application of the Li‐CO2 batteries. Herein, for the first time, we report a photo‐induced flexible Li‐CO2 battery with well‐designed, hierarchical porous, and free‐standing In2S3 @CNT/SS (ICS) as a bifunctional photoelectrode to accelerate both CDRR and CDER. The photo‐induced Li‐CO2 battery has achieved a record‐high discharge voltage of 3.14 V which surpasses the thermodynamic limit of 2.80 V, and an ultra‐low charge voltage of 3.20 V, achieving a round trip efficiency of 98.1%, which is the highest value ever reported (

https://ift.tt/2ZhZaF5

👍︎ 2
💬︎
📅︎ May 18 2020
🚨︎ report

Please note that this site uses cookies to personalise content and adverts, to provide social media features, and to analyse web traffic. Click here for more information.